Events

Time	Namespace	Component	RelatedObject	Reason	Message
16:19:08	kserve-ci-e2e-test		isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9 to ip-10-0-140-157.ec2.internal
16:30:33	kserve-ci-e2e-test		isvc-pmml-runtime-predictor-885fc8777-txjc6	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6 to ip-10-0-140-157.ec2.internal
16:29:54	kserve-ci-e2e-test		isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp to ip-10-0-128-144.ec2.internal
16:23:27	kserve-ci-e2e-test		isvc-sklearn-s3-predictor-547bfff996-2r8wd	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd to ip-10-0-128-144.ec2.internal
16:28:10	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr to ip-10-0-140-157.ec2.internal
16:26:39	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t to ip-10-0-128-144.ec2.internal
16:26:28	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng to ip-10-0-140-157.ec2.internal
16:24:57	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8 to ip-10-0-140-157.ec2.internal
16:29:43	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph to ip-10-0-128-144.ec2.internal
16:28:22	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng to ip-10-0-128-144.ec2.internal
16:09:56	kserve-ci-e2e-test		isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw to ip-10-0-128-144.ec2.internal
16:15:23	kserve-ci-e2e-test		isvc-xgboost-predictor-7566d65bf7-q4ck9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9 to ip-10-0-140-157.ec2.internal
16:08:21	kserve-ci-e2e-test		isvc-sklearn-v2-predictor-68f4f4d797-c6nvr	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr to ip-10-0-140-157.ec2.internal
16:28:36	kserve-ci-e2e-test		isvc-pmml-predictor-8674795f59-tq9pm	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm to ip-10-0-140-157.ec2.internal
16:07:15	kserve-ci-e2e-test		isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94 to ip-10-0-140-157.ec2.internal
16:27:29	kserve-ci-e2e-test		isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4 to ip-10-0-140-157.ec2.internal
16:26:13	kserve-ci-e2e-test		isvc-paddle-runtime-predictor-58b99fff76-kt7kv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv to ip-10-0-128-144.ec2.internal
16:11:31	kserve-ci-e2e-test		isvc-tensorflow-predictor-547dfc5669-9j8q7	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7 to ip-10-0-128-144.ec2.internal
15:54:52	kserve-ci-e2e-test		isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92 to ip-10-0-128-144.ec2.internal
16:24:56	kserve-ci-e2e-test		isvc-paddle-predictor-6d5c9fd4f6-pzrq5	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5 to ip-10-0-140-157.ec2.internal
16:12:19	kserve-ci-e2e-test		isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv to ip-10-0-140-157.ec2.internal
15:59:38	kserve-ci-e2e-test		isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j to ip-10-0-140-157.ec2.internal
16:05:28	kserve-ci-e2e-test		isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk to ip-10-0-128-144.ec2.internal
16:22:35	kserve-ci-e2e-test		isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2 to ip-10-0-140-157.ec2.internal
15:52:46	kserve-ci-e2e-test		isvc-predictive-xgboost-predictor-695f978978-qwkms	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms to ip-10-0-128-144.ec2.internal
15:50:25	kserve-ci-e2e-test		isvc-sklearn-batcher-predictor-6c74cc89d5-km88c	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c to ip-10-0-140-157.ec2.internal
16:08:41	kserve-ci-e2e-test		isvc-canary-promote-predictor-8499c568cb-ljpgg	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg to ip-10-0-140-157.ec2.internal
16:08:41	kserve-ci-e2e-test		isvc-canary-promote-predictor-8499c568cb-s5bz7	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7 to ip-10-0-128-144.ec2.internal
16:18:04	kserve-ci-e2e-test		xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw	Scheduled	Successfully assigned kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw to ip-10-0-140-157.ec2.internal
16:10:11	kserve-ci-e2e-test		isvc-canary-promote-v2-predictor-5654b4fcff-rq467	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467 to ip-10-0-140-157.ec2.internal
16:06:56	kserve-ci-e2e-test		isvc-canary-create-v2-predictor-b59dcb89c-4968k	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k to ip-10-0-128-144.ec2.internal
15:54:06	kserve-ci-e2e-test		isvc-logger-predictor-76485c66c8-92wfn	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn to ip-10-0-140-157.ec2.internal
16:13:06	kserve-ci-e2e-test		isvc-triton-predictor-5d7db58676-zwxtg	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg to ip-10-0-128-144.ec2.internal
15:50:40	kserve-ci-e2e-test		isvc-predictive-sklearn-predictor-5f849d69df-sswfz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz to ip-10-0-140-157.ec2.internal
16:00:54	kserve-ci-e2e-test		isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq to ip-10-0-128-144.ec2.internal
16:06:15	kserve-ci-e2e-test		isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj to ip-10-0-128-144.ec2.internal
16:15:22	kserve-ci-e2e-test		isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l to ip-10-0-128-144.ec2.internal
15:50:25	kserve-ci-e2e-test		isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff to ip-10-0-140-157.ec2.internal
15:58:33	kserve-ci-e2e-test		isvc-autogluon-v2-predictor-9684b4cd9-8zfzx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx to ip-10-0-128-144.ec2.internal
16:08:41	kserve-ci-e2e-test		isvc-canary-promote-v2-predictor-5654b4fcff-vws2k	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k to ip-10-0-128-144.ec2.internal
15:52:21	kserve-ci-e2e-test		isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v to ip-10-0-140-157.ec2.internal
16:17:02	kserve-ci-e2e-test		isvc-lightgbm-predictor-54f85ddf88-rl5dc	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc to ip-10-0-128-144.ec2.internal
16:13:17	kserve-ci-e2e-test		isvc-canary-rollback-predictor-78f9bdc99b-9vph6	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6 to ip-10-0-128-144.ec2.internal
16:03:23	kserve-ci-e2e-test		sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k	Scheduled	Successfully assigned kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k to ip-10-0-140-157.ec2.internal
15:55:41	kserve-ci-e2e-test		isvc-autogluon-v1-predictor-574b5b7495-7j94v	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v to ip-10-0-128-144.ec2.internal
16:15:22	kserve-ci-e2e-test		isvc-canary-stop-predictor-556d599d48-nhvnp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp to ip-10-0-140-157.ec2.internal
15:58:03	kserve-ci-e2e-test		isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2 to ip-10-0-140-157.ec2.internal
16:11:47	kserve-ci-e2e-test		isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f to ip-10-0-128-144.ec2.internal
16:32:20	kserve-ci-e2e-test		isvc-primary-859eae-predictor-684899d844-qpnjm	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm to ip-10-0-128-144.ec2.internal
16:01:48	kserve-ci-e2e-test		isvc-sklearn-predictor-58f64f5dbf-hqww4	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4 to ip-10-0-140-157.ec2.internal
16:21:05	kserve-ci-e2e-test		isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w to ip-10-0-140-157.ec2.internal
16:06:38	kserve-ci-e2e-test		isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5 to ip-10-0-140-157.ec2.internal
16:15:22	kserve-ci-e2e-test		isvc-canary-stop-predictor-556d599d48-pcrdx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx to ip-10-0-128-144.ec2.internal
16:23:31	kserve-ci-e2e-test		isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj to ip-10-0-128-144.ec2.internal
16:06:56	kserve-ci-e2e-test		isvc-canary-create-predictor-5674b8c456-rd6t5	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5 to ip-10-0-140-157.ec2.internal
16:11:47	kserve-ci-e2e-test		isvc-canary-rollback-predictor-78f9bdc99b-kvk8k	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k to ip-10-0-140-157.ec2.internal
16:06:56	kserve-ci-e2e-test		isvc-canary-create-predictor-5674b8c456-f6s8t	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t to ip-10-0-128-144.ec2.internal
16:03:16	kserve-ci-e2e-test		isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht to ip-10-0-128-144.ec2.internal
15:56:28	kserve-ci-e2e-test		isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79 to ip-10-0-140-157.ec2.internal
16:22:00	kserve-ci-e2e-test		isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx to ip-10-0-128-144.ec2.internal
16:19:00	kserve-ci-e2e-test		isvc-xgboost-runtime-predictor-54dd488c85-tl8h4	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4 to ip-10-0-128-144.ec2.internal
16:17:09	kserve-ci-e2e-test		isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9 to ip-10-0-140-157.ec2.internal
16:22:01	kserve-ci-e2e-test		isvc-xgboost-v2-predictor-fcc668bc-tdbqd	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd to ip-10-0-140-157.ec2.internal
16:20:56	kserve-ci-e2e-test		isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk to ip-10-0-140-157.ec2.internal
15:53:46	kserve-ci-e2e-test		message-dumper-predictor-6575bcfdfd-5kzf7	Scheduled	Successfully assigned kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7 to ip-10-0-140-157.ec2.internal
16:34:05	kserve-ci-e2e-test		isvc-secondary-859eae-predictor-54d5c46d5f-blvk6	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6 to ip-10-0-140-157.ec2.internal
16:11:47	kserve-ci-e2e-test		isvc-canary-rollback-predictor-78f9bdc99b-bgv79	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79 to ip-10-0-128-144.ec2.internal
15:50:25	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-predictor-6c74cc89d5	SuccessfulCreate	Created pod: isvc-sklearn-batcher-predictor-6c74cc89d5-km88c
15:50:25	kserve-ci-e2e-test	deployment-controller	isvc-init-fail-f9f2bc-predictor	ScalingReplicaSet	Scaled up replica set isvc-init-fail-f9f2bc-predictor-85cb9fd88f from 0 to 1
15:50:25	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
15:50:25	kserve-ci-e2e-test	replicaset-controller	isvc-init-fail-f9f2bc-predictor-85cb9fd88f	SuccessfulCreate	Created pod: isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff
15:50:25	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-predictor-6c74cc89d5 from 0 to 1
15:50:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c74cc89d5-km88c	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
15:50:26	kserve-ci-e2e-test	kubelet	isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-init-fail-f9f2bc-predictor-serving-cert" not found
15:50:26	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-predictor-6c74cc89d5-km88c	AddedInterface	Add eth0 [10.132.0.24/23] from ovn-kubernetes
15:50:27	kserve-ci-e2e-test	multus	isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff	AddedInterface	Add eth0 [10.132.0.25/23] from ovn-kubernetes
15:50:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c74cc89d5-km88c	Started	Started container storage-initializer
15:50:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c74cc89d5-km88c	Created	Created container: storage-initializer
15:50:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c74cc89d5-km88c	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
15:50:31 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff	Created	Created container: storage-initializer
15:50:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c74cc89d5-km88c	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996"
15:50:31 (x8)	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-f9f2bc	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:50:31 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
15:50:31 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff	Started	Started container storage-initializer
15:50:35	kserve-ci-e2e-test	kubelet	isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff	Killing	Stopping container storage-initializer
15:50:40	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-predictor-5f849d69df from 0 to 1
15:50:40	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-predictor-5f849d69df	SuccessfulCreate	Created pod: isvc-predictive-sklearn-predictor-5f849d69df-sswfz
15:50:40 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn": the object has been modified; please apply your changes to the latest version and try again
15:50:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn": the object has been modified; please apply your changes to the latest version and try again
15:50:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c74cc89d5-km88c	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
15:50:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c74cc89d5-km88c	Created	Created container: kserve-container
15:50:44	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-sswfz	Started	Started container storage-initializer
15:50:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c74cc89d5-km88c	Started	Started container kserve-container
15:50:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c74cc89d5-km88c	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" in 12.922s (12.922s including waiting). Image size: 1580836240 bytes.
15:50:44	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-sswfz	Created	Created container: storage-initializer
15:50:44	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-sswfz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
15:50:44	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-predictor-5f849d69df-sswfz	AddedInterface	Add eth0 [10.132.0.26/23] from ovn-kubernetes
15:50:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c74cc89d5-km88c	Created	Created container: kube-rbac-proxy
15:50:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c74cc89d5-km88c	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.152s (2.152s including waiting). Image size: 211946088 bytes.
15:50:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c74cc89d5-km88c	Started	Started container kube-rbac-proxy
15:50:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c74cc89d5-km88c	Pulling	Pulling image "quay.io/opendatahub/kserve-agent@sha256:9551f4f790f8c157387199c857354f4c235156d40ea86876db013e8cd552e398"
15:50:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c74cc89d5-km88c	Started	Started container agent
15:50:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c74cc89d5-km88c	Created	Created container: agent
15:50:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c74cc89d5-km88c	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:9551f4f790f8c157387199c857354f4c235156d40ea86876db013e8cd552e398" in 2.17s (2.17s including waiting). Image size: 240302150 bytes.
15:50:49	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-sswfz	Pulling	Pulling image "kserve/predictiveserver:latest"
15:51:11	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-sswfz	Created	Created container: kserve-container
15:51:11	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-sswfz	Started	Started container kube-rbac-proxy
15:51:11	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-sswfz	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
15:51:11	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-sswfz	Pulled	Successfully pulled image "kserve/predictiveserver:latest" in 22.039s (22.039s including waiting). Image size: 2438904977 bytes.
15:51:11	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-sswfz	Started	Started container kserve-container
15:51:11	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-sswfz	Created	Created container: kube-rbac-proxy
15:51:25 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:51:25 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:51:40 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:51:40 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:52:10 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:52:10 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:52:12 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:52:12	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InferenceServiceReady	InferenceService [isvc-sklearn-batcher] is Ready
15:52:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c74cc89d5-km88c	Killing	Stopping container agent
15:52:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c74cc89d5-km88c	Killing	Stopping container kserve-container
15:52:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c74cc89d5-km88c	Killing	Stopping container kube-rbac-proxy
15:52:20	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-custom-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-custom-predictor-76bd9f8cf9 from 0 to 1
15:52:21	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-custom-predictor-76bd9f8cf9	SuccessfulCreate	Created pod: isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v
15:52:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
15:52:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v	Created	Created container: storage-initializer
15:52:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v	Started	Started container storage-initializer
15:52:21	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v	AddedInterface	Add eth0 [10.132.0.27/23] from ovn-kubernetes
15:52:22 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c74cc89d5-km88c	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
15:52:22 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c74cc89d5-km88c	Unhealthy	Readiness probe failed: dial tcp 10.132.0.24:8080: connect: connection refused
15:52:25	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" already present on machine
15:52:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v	Started	Started container kserve-container
15:52:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v	Created	Created container: kube-rbac-proxy
15:52:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v	Started	Started container kube-rbac-proxy
15:52:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v	Started	Started container agent
15:52:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v	Created	Created container: kserve-container
15:52:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v	Created	Created container: agent
15:52:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:9551f4f790f8c157387199c857354f4c235156d40ea86876db013e8cd552e398" already present on machine
15:52:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
15:52:29 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-sswfz	Unhealthy	Readiness probe failed: dial tcp 10.132.0.26:8080: connect: connection refused
15:52:30 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c74cc89d5-km88c	Unhealthy	Readiness probe failed: Get "https://10.132.0.24:8643/healthz": dial tcp 10.132.0.24:8643: connect: connection refused
15:52:39 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:52:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	InferenceServiceReady	InferenceService [isvc-predictive-sklearn] is Ready
15:52:40 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:52:40 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:52:41	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-sswfz	Killing	Stopping container kserve-container
15:52:41	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-sswfz	Killing	Stopping container kube-rbac-proxy
15:52:44	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-sswfz	Unhealthy	Readiness probe failed: Get "https://10.132.0.26:8643/healthz": dial tcp 10.132.0.26:8643: connect: connection refused
15:52:46	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-predictor-695f978978 from 0 to 1
15:52:46	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost": the object has been modified; please apply your changes to the latest version and try again
15:52:46	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-predictor-695f978978	SuccessfulCreate	Created pod: isvc-predictive-xgboost-predictor-695f978978-qwkms
15:52:47	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-qwkms	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4"
15:52:47	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-predictor-695f978978-qwkms	AddedInterface	Add eth0 [10.133.0.23/23] from ovn-kubernetes
15:52:50	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-qwkms	Created	Created container: storage-initializer
15:52:50	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-qwkms	Started	Started container storage-initializer
15:52:50	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-qwkms	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" in 3.491s (3.491s including waiting). Image size: 306739622 bytes.
15:52:50 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:52:50 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:52:55	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-qwkms	Pulling	Pulling image "kserve/predictiveserver:latest"
15:53:17	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-qwkms	Created	Created container: kserve-container
15:53:17	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-qwkms	Pulled	Successfully pulled image "kserve/predictiveserver:latest" in 22.585s (22.585s including waiting). Image size: 2438904977 bytes.
15:53:17	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-qwkms	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
15:53:17	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-qwkms	Started	Started container kserve-container
15:53:20	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-qwkms	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.926s (2.926s including waiting). Image size: 211946088 bytes.
15:53:20	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-qwkms	Created	Created container: kube-rbac-proxy
15:53:20	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-qwkms	Started	Started container kube-rbac-proxy
15:53:31 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:53:31 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:53:36 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:53:36 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:53:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	InferenceServiceReady	InferenceService [isvc-sklearn-batcher-custom] is Ready
15:53:40 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:53:46	kserve-ci-e2e-test	replicaset-controller	message-dumper-predictor-6575bcfdfd	SuccessfulCreate	Created pod: message-dumper-predictor-6575bcfdfd-5kzf7
15:53:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v	Killing	Stopping container kserve-container
15:53:46	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	UpdateFailed	Failed to update status for InferenceService "message-dumper": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper": the object has been modified; please apply your changes to the latest version and try again
15:53:46	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-5kzf7	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "message-dumper-predictor-serving-cert" not found
15:53:46	kserve-ci-e2e-test	deployment-controller	message-dumper-predictor	ScalingReplicaSet	Scaled up replica set message-dumper-predictor-6575bcfdfd from 0 to 1
15:53:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v	Killing	Stopping container kube-rbac-proxy
15:53:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v	Killing	Stopping container agent
15:53:47	kserve-ci-e2e-test	multus	message-dumper-predictor-6575bcfdfd-5kzf7	AddedInterface	Add eth0 [10.132.0.28/23] from ovn-kubernetes
15:53:47	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-5kzf7	Pulling	Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display"
15:53:48	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-5kzf7	Created	Created container: kserve-container
15:53:48	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-5kzf7	Started	Started container kserve-container
15:53:48	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-5kzf7	Pulled	Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.35s (1.35s including waiting). Image size: 14813193 bytes.
15:53:48	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-5kzf7	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
15:53:48	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-5kzf7	Started	Started container kube-rbac-proxy
15:53:48	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-5kzf7	Created	Created container: kube-rbac-proxy
15:53:56 (x8)	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:53:56	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	InferenceServiceReady	InferenceService [message-dumper] is Ready
15:53:57 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v	Unhealthy	Readiness probe failed: Get "https://10.132.0.27:8643/healthz": dial tcp 10.132.0.27:8643: connect: connection refused
15:54:00 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
15:54:00 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v	Unhealthy	Readiness probe failed: dial tcp 10.132.0.27:5000: connect: connection refused
15:54:06	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-76485c66c8-92wfn	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
15:54:06	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-76485c66c8-92wfn	Created	Created container: storage-initializer
15:54:06	kserve-ci-e2e-test	multus	isvc-logger-predictor-76485c66c8-92wfn	AddedInterface	Add eth0 [10.132.0.29/23] from ovn-kubernetes
15:54:06	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-76485c66c8-92wfn	Started	Started container storage-initializer
15:54:06	kserve-ci-e2e-test	replicaset-controller	isvc-logger-predictor-76485c66c8	SuccessfulCreate	Created pod: isvc-logger-predictor-76485c66c8-92wfn
15:54:06	kserve-ci-e2e-test	deployment-controller	isvc-logger-predictor	ScalingReplicaSet	Scaled up replica set isvc-logger-predictor-76485c66c8 from 0 to 1
15:54:06	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	UpdateFailed	Failed to update status for InferenceService "isvc-logger": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger": the object has been modified; please apply your changes to the latest version and try again
15:54:11	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-76485c66c8-92wfn	Started	Started container kube-rbac-proxy
15:54:11	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-76485c66c8-92wfn	Created	Created container: agent
15:54:11	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-76485c66c8-92wfn	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" already present on machine
15:54:11	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-76485c66c8-92wfn	Started	Started container agent
15:54:11	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-76485c66c8-92wfn	Created	Created container: kserve-container
15:54:11	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-76485c66c8-92wfn	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:9551f4f790f8c157387199c857354f4c235156d40ea86876db013e8cd552e398" already present on machine
15:54:11	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-76485c66c8-92wfn	Started	Started container kserve-container
15:54:11	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-76485c66c8-92wfn	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
15:54:11	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-76485c66c8-92wfn	Created	Created container: kube-rbac-proxy
15:54:16	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:54:16	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:54:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:54:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:54:36 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:54:36 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:54:38 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:54:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	InferenceServiceReady	InferenceService [isvc-predictive-xgboost] is Ready
15:54:46 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:54:46 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:54:47	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-qwkms	Killing	Stopping container kserve-container
15:54:47	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-qwkms	Killing	Stopping container kube-rbac-proxy
15:54:48	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-qwkms	Unhealthy	Readiness probe failed: Get "https://10.133.0.23:8643/healthz": dial tcp 10.133.0.23:8643: connect: connection refused
15:54:48 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-qwkms	Unhealthy	Readiness probe failed: dial tcp 10.133.0.23:8080: connect: connection refused
15:54:52	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm": the object has been modified; please apply your changes to the latest version and try again
15:54:52	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-predictor-ff56bcbb5 from 0 to 1
15:54:52	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-predictor-ff56bcbb5	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92
15:54:52	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92	Started	Started container storage-initializer
15:54:52	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92	Created	Created container: storage-initializer
15:54:52	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
15:54:52	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92	AddedInterface	Add eth0 [10.133.0.24/23] from ovn-kubernetes
15:54:57	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92	Created	Created container: kube-rbac-proxy
15:54:57	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92	Started	Started container kube-rbac-proxy
15:54:57	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
15:54:57	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92	Started	Started container kserve-container
15:54:57	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92	Created	Created container: kserve-container
15:54:57	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
15:55:07	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:55:07	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:55:28 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:55:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	InferenceServiceReady	InferenceService [isvc-logger] is Ready
15:55:36 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:55:36 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:55:41	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-76485c66c8-92wfn	Killing	Stopping container kube-rbac-proxy
15:55:41	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v1-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v1-predictor-574b5b7495 from 0 to 1
15:55:41	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-5kzf7	Killing	Stopping container kserve-container
15:55:41	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v1-predictor-574b5b7495	SuccessfulCreate	Created pod: isvc-autogluon-v1-predictor-574b5b7495-7j94v
15:55:41	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-5kzf7	Killing	Stopping container kube-rbac-proxy
15:55:41	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-76485c66c8-92wfn	Killing	Stopping container kserve-container
15:55:41	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-76485c66c8-92wfn	Killing	Stopping container agent
15:55:42	kserve-ci-e2e-test	multus	isvc-autogluon-v1-predictor-574b5b7495-7j94v	AddedInterface	Add eth0 [10.133.0.25/23] from ovn-kubernetes
15:55:42	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-7j94v	Started	Started container storage-initializer
15:55:42	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-7j94v	Created	Created container: storage-initializer
15:55:42	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-7j94v	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
15:55:48 (x10)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-76485c66c8-92wfn	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
15:55:49	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-7j94v	Pulling	Pulling image "kserve/autogluonserver:latest"
15:55:58 (x4)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-76485c66c8-92wfn	Unhealthy	Readiness probe failed: Get "https://10.132.0.29:8643/healthz": dial tcp 10.132.0.29:8643: connect: connection refused
15:55:58 (x11)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-76485c66c8-92wfn	Unhealthy	Readiness probe failed: dial tcp 10.132.0.29:8080: connect: connection refused
15:56:15 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92	Unhealthy	Readiness probe failed: dial tcp 10.133.0.24:8080: connect: connection refused
15:56:21 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:56:21	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm] is Ready
15:56:22 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:56:22 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:56:23	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92	Killing	Stopping container kube-rbac-proxy
15:56:23	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92	Killing	Stopping container kserve-container
15:56:25	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92	Unhealthy	Readiness probe failed: Get "https://10.133.0.24:8643/healthz": dial tcp 10.133.0.24:8643: connect: connection refused
15:56:28	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-v2-predictor-745f9c8c9f from 0 to 1
15:56:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
15:56:28	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79	Created	Created container: storage-initializer
15:56:28	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
15:56:28	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79	AddedInterface	Add eth0 [10.132.0.30/23] from ovn-kubernetes
15:56:28	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79	Started	Started container storage-initializer
15:56:28	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-v2-predictor-745f9c8c9f	SuccessfulCreate	Created pod: isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79
15:56:33	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
15:56:33	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79	Started	Started container kube-rbac-proxy
15:56:33	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79	Created	Created container: kube-rbac-proxy
15:56:33	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79	Started	Started container kserve-container
15:56:33	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79	Created	Created container: kserve-container
15:56:33	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
15:57:13 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:57:13 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:57:51 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:57:51	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	InferenceServiceReady	InferenceService [isvc-predictive-sklearn-v2] is Ready
15:57:55	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-7j94v	Started	Started container kube-rbac-proxy
15:57:55	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-7j94v	Pulled	Successfully pulled image "kserve/autogluonserver:latest" in 2m5.698s (2m5.698s including waiting). Image size: 8948761381 bytes.
15:57:55	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-7j94v	Created	Created container: kserve-container
15:57:55	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-7j94v	Started	Started container kserve-container
15:57:55	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-7j94v	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
15:57:55	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-7j94v	Created	Created container: kube-rbac-proxy
15:57:58 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:57:58	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79	Killing	Stopping container kserve-container
15:57:58 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:57:58	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79	Killing	Stopping container kube-rbac-proxy
15:58:01 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79	Unhealthy	Readiness probe failed: Get "http://10.132.0.30:8080/v2/models/isvc-predictive-sklearn-v2/ready": dial tcp 10.132.0.30:8080: connect: connection refused
15:58:01	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79	Unhealthy	Readiness probe failed: Get "https://10.132.0.30:8643/healthz": dial tcp 10.132.0.30:8643: connect: connection refused
15:58:03	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-v2-predictor-868ff5dd6d	SuccessfulCreate	Created pod: isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2
15:58:03 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-7j94v	Unhealthy	Readiness probe failed: dial tcp 10.133.0.25:8080: connect: connection refused
15:58:03 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
15:58:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
15:58:03	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-v2-predictor-868ff5dd6d from 0 to 1
15:58:04	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2	Started	Started container storage-initializer
15:58:04	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2	Created	Created container: storage-initializer
15:58:04	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
15:58:04	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2	AddedInterface	Add eth0 [10.132.0.31/23] from ovn-kubernetes
15:58:08	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2	Created	Created container: kserve-container
15:58:08	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
15:58:08	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2	Started	Started container kube-rbac-proxy
15:58:08	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2	Created	Created container: kube-rbac-proxy
15:58:08	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
15:58:08	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2	Started	Started container kserve-container
15:58:11 (x10)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:58:11 (x10)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:58:13 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:58:13	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	InferenceServiceReady	InferenceService [isvc-autogluon-v1] is Ready
15:58:26	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:58:26	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:58:27	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-7j94v	Killing	Stopping container kserve-container
15:58:27	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-7j94v	Killing	Stopping container kube-rbac-proxy
15:58:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-7j94v	Unhealthy	Readiness probe failed: Get "https://10.133.0.25:8643/healthz": dial tcp 10.133.0.25:8643: connect: connection refused
15:58:32	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2": the object has been modified; please apply your changes to the latest version and try again
15:58:32	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-predictor-9684b4cd9 from 0 to 1
15:58:33	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-predictor-9684b4cd9	SuccessfulCreate	Created pod: isvc-autogluon-v2-predictor-9684b4cd9-8zfzx
15:58:33 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:58:33 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:58:33	kserve-ci-e2e-test	multus	isvc-autogluon-v2-predictor-9684b4cd9-8zfzx	AddedInterface	Add eth0 [10.133.0.26/23] from ovn-kubernetes
15:58:33	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-8zfzx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
15:58:33	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-8zfzx	Created	Created container: storage-initializer
15:58:33	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-8zfzx	Started	Started container storage-initializer
15:59:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-8zfzx	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
15:59:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-8zfzx	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
15:59:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-8zfzx	Created	Created container: kube-rbac-proxy
15:59:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-8zfzx	Created	Created container: kserve-container
15:59:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-8zfzx	Started	Started container kube-rbac-proxy
15:59:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-8zfzx	Started	Started container kserve-container
15:59:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:59:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:59:28 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:59:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	InferenceServiceReady	InferenceService [isvc-predictive-xgboost-v2] is Ready
15:59:33 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:59:33 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:59:33	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2	Killing	Stopping container kube-rbac-proxy
15:59:33	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2	Killing	Stopping container kserve-container
15:59:35	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2	Unhealthy	Readiness probe failed: Get "https://10.132.0.31:8643/healthz": dial tcp 10.132.0.31:8643: connect: connection refused
15:59:37 (x6)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2	Unhealthy	Readiness probe failed: Get "http://10.132.0.31:8080/v2/models/isvc-predictive-xgboost-v2/ready": dial tcp 10.132.0.31:8080: connect: connection refused
15:59:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm-v2": the object has been modified; please apply your changes to the latest version and try again
15:59:38 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm-v2": the object has been modified; please apply your changes to the latest version and try again
15:59:38	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-v2-predictor-545b7cc587 from 0 to 1
15:59:38	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-v2-predictor-545b7cc587	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j
15:59:40	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j	AddedInterface	Add eth0 [10.132.0.32/23] from ovn-kubernetes
15:59:40	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j	Started	Started container storage-initializer
15:59:40	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
15:59:40	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j	Created	Created container: storage-initializer
15:59:40	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : failed to sync secret cache: timed out waiting for the condition
15:59:40	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j	FailedMount	MountVolume.SetUp failed for volume "kube-rbac-proxy-sar-config" : failed to sync configmap cache: timed out waiting for the condition
15:59:45	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
15:59:45	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j	Started	Started container kserve-container
15:59:45	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j	Created	Created container: kserve-container
15:59:45	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
15:59:46	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j	Created	Created container: kube-rbac-proxy
15:59:46	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j	Started	Started container kube-rbac-proxy
16:00:23 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:00:23 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:00:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	InferenceServiceReady	InferenceService [isvc-autogluon-v2] is Ready
16:00:34 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:00:45	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-8zfzx	Unhealthy	Readiness probe failed: Get "http://10.133.0.26:8080/v2/models/isvc-autogluon-v2/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
16:00:48 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:00:48 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:00:49	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-8zfzx	Killing	Stopping container kube-rbac-proxy
16:00:49	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-8zfzx	Killing	Stopping container kserve-container
16:00:54	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-variants-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-variants-predictor-5c889d8844 from 0 to 1
16:00:54	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
16:00:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2-variants": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-variants": the object has been modified; please apply your changes to the latest version and try again
16:00:54	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-variants-predictor-5c889d8844	SuccessfulCreate	Created pod: isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq
16:00:55	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq	Created	Created container: storage-initializer
16:00:55	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq	Started	Started container storage-initializer
16:00:55	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:00:55	kserve-ci-e2e-test	multus	isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq	AddedInterface	Add eth0 [10.133.0.27/23] from ovn-kubernetes
16:01:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm-v2] is Ready
16:01:04 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:01:08 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:01:08	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j	Killing	Stopping container kube-rbac-proxy
16:01:08	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j	Killing	Stopping container kserve-container
16:01:08 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:01:13 (x2)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j	Unhealthy	Readiness probe failed: Get "https://10.132.0.32:8643/healthz": dial tcp 10.132.0.32:8643: connect: connection refused
16:01:13 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j	Unhealthy	Readiness probe failed: Get "http://10.132.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready": dial tcp 10.132.0.32:8080: connect: connection refused
16:01:14	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-scheduler-predictor-546d99c7cc	SuccessfulCreate	Created pod: isvc-sklearn-scheduler-predictor-546d99c7cc-hzt8b
16:01:14 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-scheduler": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
16:01:14 (x6)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:01:14 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
16:01:14	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-scheduler-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-scheduler-predictor-546d99c7cc from 0 to 1
16:01:26	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:01:26	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq	Started	Started container kserve-container
16:01:26	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq	Created	Created container: kube-rbac-proxy
16:01:26	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
16:01:26	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq	Created	Created container: kserve-container
16:01:26	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq	Started	Started container kube-rbac-proxy
16:01:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn": the object has been modified; please apply your changes to the latest version and try again
16:01:47	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-predictor-58f64f5dbf from 0 to 1
16:01:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-58f64f5dbf-hqww4	Created	Created container: storage-initializer
16:01:48	kserve-ci-e2e-test	multus	isvc-sklearn-predictor-58f64f5dbf-hqww4	AddedInterface	Add eth0 [10.132.0.35/23] from ovn-kubernetes
16:01:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-58f64f5dbf-hqww4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:01:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-58f64f5dbf-hqww4	Started	Started container storage-initializer
16:01:48	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-predictor-58f64f5dbf	SuccessfulCreate	Created pod: isvc-sklearn-predictor-58f64f5dbf-hqww4
16:01:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-58f64f5dbf-hqww4	Created	Created container: kserve-container
16:01:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-58f64f5dbf-hqww4	Started	Started container kube-rbac-proxy
16:01:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-58f64f5dbf-hqww4	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" already present on machine
16:01:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-58f64f5dbf-hqww4	Started	Started container kserve-container
16:01:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-58f64f5dbf-hqww4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:01:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-58f64f5dbf-hqww4	Created	Created container: kube-rbac-proxy
16:01:54 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:01:54 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:02:17 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:02:17 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:03:02 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:03:02	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	InferenceServiceReady	InferenceService [isvc-autogluon-v2-variants] is Ready
16:03:09 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:03:09 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:03:11 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:03:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	InferenceServiceReady	InferenceService [isvc-sklearn] is Ready
16:03:11	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq	Killing	Stopping container kserve-container
16:03:11	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq	Killing	Stopping container kube-rbac-proxy
16:03:12	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq	Unhealthy	Readiness probe failed: Get "http://10.133.0.27:8080/v2/models/isvc-autogluon-v2-variants/ready": dial tcp 10.133.0.27:8080: connect: connection refused
16:03:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2-noslash": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-noslash": the object has been modified; please apply your changes to the latest version and try again
16:03:16	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-noslash-predictor-5499c79f9f	SuccessfulCreate	Created pod: isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht
16:03:16	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-noslash-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-noslash-predictor-5499c79f9f from 0 to 1
16:03:17	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht	Started	Started container storage-initializer
16:03:17	kserve-ci-e2e-test	multus	isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht	AddedInterface	Add eth0 [10.133.0.28/23] from ovn-kubernetes
16:03:17	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:03:17	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht	Created	Created container: storage-initializer
16:03:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-58f64f5dbf-hqww4	Killing	Stopping container kserve-container
16:03:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-58f64f5dbf-hqww4	Killing	Stopping container kube-rbac-proxy
16:03:18 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:03:18 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:03:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-58f64f5dbf-hqww4	Unhealthy	Readiness probe failed: Get "https://10.132.0.35:8643/healthz": dial tcp 10.132.0.35:8643: connect: connection refused
16:03:21 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-58f64f5dbf-hqww4	Unhealthy	Readiness probe failed: dial tcp 10.132.0.35:8080: connect: connection refused
16:03:23	kserve-ci-e2e-test	replicaset-controller	sklearn-v2-mlserver-predictor-7f568cdc6b	SuccessfulCreate	Created pod: sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k
16:03:23	kserve-ci-e2e-test	multus	sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k	AddedInterface	Add eth0 [10.132.0.36/23] from ovn-kubernetes
16:03:23	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:03:23	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k	Created	Created container: storage-initializer
16:03:23	kserve-ci-e2e-test	deployment-controller	sklearn-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set sklearn-v2-mlserver-predictor-7f568cdc6b from 0 to 1
16:03:23	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "sklearn-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
16:03:23 (x2)	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "sklearn-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "sklearn-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
16:03:24	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k	Started	Started container storage-initializer
16:03:28	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k	Pulling	Pulling image "docker.io/seldonio/mlserver:1.7.1"
16:03:38	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:03:38	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht	Created	Created container: kube-rbac-proxy
16:03:38	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht	Created	Created container: kserve-container
16:03:38	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht	Started	Started container kube-rbac-proxy
16:03:38	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
16:03:38	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht	Started	Started container kserve-container
16:04:16 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:04:16 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:05:15 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:05:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	InferenceServiceReady	InferenceService [isvc-autogluon-v2-noslash] is Ready
16:05:16 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:05:16 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:05:23	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht	Killing	Stopping container kube-rbac-proxy
16:05:23	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht	Killing	Stopping container kserve-container
16:05:28	kserve-ci-e2e-test	multus	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk	AddedInterface	Add eth0 [10.133.0.29/23] from ovn-kubernetes
16:05:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-ts-v1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1": the object has been modified; please apply your changes to the latest version and try again
16:05:28	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-ts-v1-predictor-6cf79d9f7d	SuccessfulCreate	Created pod: isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk
16:05:28	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-ts-v1-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-ts-v1-predictor-6cf79d9f7d from 0 to 1
16:05:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk	Created	Created container: storage-initializer
16:05:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:05:29	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk	Started	Started container storage-initializer
16:05:39	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk	Created	Created container: kserve-container
16:05:39	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk	Started	Started container kube-rbac-proxy
16:05:39	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
16:05:39	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk	Created	Created container: kube-rbac-proxy
16:05:39	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:05:39	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk	Started	Started container kserve-container
16:05:41	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k	Created	Created container: kube-rbac-proxy
16:05:41	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k	Started	Started container kserve-container
16:05:41	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k	Created	Created container: kserve-container
16:05:41	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k	Pulled	Successfully pulled image "docker.io/seldonio/mlserver:1.7.1" in 2m12.209s (2m12.209s including waiting). Image size: 10890461297 bytes.
16:05:41	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:05:41	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k	Started	Started container kube-rbac-proxy
16:05:46 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk	Unhealthy	Readiness probe failed: dial tcp 10.133.0.29:8080: connect: connection refused
16:05:56 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:05:56	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	InferenceServiceReady	InferenceService [isvc-autogluon-ts-v1] is Ready
16:05:58 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:05:58 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:06:08 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:06:08 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:06:10	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk	Killing	Stopping container kserve-container
16:06:10	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk	Killing	Stopping container kube-rbac-proxy
16:06:11	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk	Unhealthy	Readiness probe failed: Get "https://10.133.0.29:8643/healthz": dial tcp 10.133.0.29:8643: connect: connection refused
16:06:15	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585	SuccessfulCreate	Created pod: isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj
16:06:15	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-ts-v1-noslash-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-ts-v1-noslash-predictor-b54fbf585 from 0 to 1
16:06:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-ts-v1-noslash": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1-noslash": the object has been modified; please apply your changes to the latest version and try again
16:06:16	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:06:16	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj	Created	Created container: storage-initializer
16:06:16	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj	Started	Started container storage-initializer
16:06:16	kserve-ci-e2e-test	multus	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj	AddedInterface	Add eth0 [10.133.0.30/23] from ovn-kubernetes
16:06:17	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
16:06:23	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:06:23	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:06:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj	Started	Started container kserve-container
16:06:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj	Started	Started container kube-rbac-proxy
16:06:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj	Created	Created container: kube-rbac-proxy
16:06:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:06:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj	Created	Created container: kserve-container
16:06:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
16:06:27	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	InferenceServiceReady	InferenceService [sklearn-v2-mlserver] is Ready
16:06:27 (x11)	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:06:32 (x4)	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj	Unhealthy	Readiness probe failed: dial tcp 10.133.0.30:8080: connect: connection refused
16:06:33	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k	Killing	Stopping container kserve-container
16:06:33	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k	Killing	Stopping container kube-rbac-proxy
16:06:37	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k	Unhealthy	Readiness probe failed: Get "https://10.132.0.36:8643/healthz": dial tcp 10.132.0.36:8643: connect: connection refused
16:06:37	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k	Unhealthy	Readiness probe failed: Get "http://10.132.0.36:8080/v2/models/sklearn-v2-mlserver/ready": dial tcp 10.132.0.36:8080: connect: connection refused
16:06:38	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-runtime-predictor-56d59b6d5c from 0 to 1
16:06:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-runtime": the object has been modified; please apply your changes to the latest version and try again
16:06:38	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-runtime-predictor-56d59b6d5c	SuccessfulCreate	Created pod: isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5
16:06:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5	Started	Started container storage-initializer
16:06:39	kserve-ci-e2e-test	multus	isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5	AddedInterface	Add eth0 [10.132.0.37/23] from ovn-kubernetes
16:06:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5	Created	Created container: storage-initializer
16:06:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:06:42 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:06:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	InferenceServiceReady	InferenceService [isvc-autogluon-ts-v1-noslash] is Ready
16:06:45 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:06:45 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:06:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5	Started	Started container kube-rbac-proxy
16:06:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" already present on machine
16:06:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5	Created	Created container: kube-rbac-proxy
16:06:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5	Created	Created container: kserve-container
16:06:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:06:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5	Started	Started container kserve-container
16:06:50	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj	Killing	Stopping container kube-rbac-proxy
16:06:50	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj	Killing	Stopping container kserve-container
16:06:53 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5	Unhealthy	Readiness probe failed: dial tcp 10.132.0.37:8080: connect: connection refused
16:06:53	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj	Unhealthy	Readiness probe failed: Get "https://10.133.0.30:8643/healthz": context deadline exceeded
16:06:53	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:06:53	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:06:56	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-f6s8t	Created	Created container: storage-initializer
16:06:56	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-5674b8c456	SuccessfulCreate	Created pod: isvc-canary-create-predictor-5674b8c456-rd6t5
16:06:56 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again
16:06:56	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-create-v2-predictor-b59dcb89c from 0 to 1
16:06:56	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-b59dcb89c-4968k	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-create-v2-predictor-serving-cert" not found
16:06:56	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-rd6t5	Started	Started container storage-initializer
16:06:56	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-f6s8t	Started	Started container storage-initializer
16:06:56	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-v2-predictor-b59dcb89c	SuccessfulCreate	Created pod: isvc-canary-create-v2-predictor-b59dcb89c-4968k
16:06:56	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-create-predictor-5674b8c456 from 0 to 2
16:06:56	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-rd6t5	Created	Created container: storage-initializer
16:06:56	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-rd6t5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:06:56	kserve-ci-e2e-test	multus	isvc-canary-create-predictor-5674b8c456-rd6t5	AddedInterface	Add eth0 [10.132.0.38/23] from ovn-kubernetes
16:06:56	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-f6s8t	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:06:56	kserve-ci-e2e-test	multus	isvc-canary-create-predictor-5674b8c456-f6s8t	AddedInterface	Add eth0 [10.133.0.32/23] from ovn-kubernetes
16:06:56	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-5674b8c456	SuccessfulCreate	Created pod: isvc-canary-create-predictor-5674b8c456-f6s8t
16:06:57	kserve-ci-e2e-test	multus	isvc-canary-create-v2-predictor-b59dcb89c-4968k	AddedInterface	Add eth0 [10.133.0.31/23] from ovn-kubernetes
16:06:57	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-b59dcb89c-4968k	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:06:57	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-b59dcb89c-4968k	Started	Started container storage-initializer
16:06:57	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-b59dcb89c-4968k	Created	Created container: storage-initializer
16:07:00	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-rd6t5	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" already present on machine
16:07:01	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-rd6t5	Created	Created container: kserve-container
16:07:01	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-rd6t5	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:07:01	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-rd6t5	Created	Created container: kube-rbac-proxy
16:07:01	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-rd6t5	Started	Started container kube-rbac-proxy
16:07:01	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-f6s8t	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996"
16:07:01	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-rd6t5	Started	Started container kserve-container
16:07:02	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-b59dcb89c-4968k	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996"
16:07:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-runtime] is Ready
16:07:04 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:07:08	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:07:08	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:07:10	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5	Killing	Stopping container kserve-container
16:07:10	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5	Killing	Stopping container kube-rbac-proxy
16:07:15	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-runtime-predictor-5f69c7d498	SuccessfulCreate	Created pod: isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94
16:07:15	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-runtime-predictor-5f69c7d498 from 0 to 1
16:07:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
16:07:16	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:07:16	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94	Started	Started container storage-initializer
16:07:16	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94	Created	Created container: storage-initializer
16:07:16	kserve-ci-e2e-test	multus	isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94	AddedInterface	Add eth0 [10.132.0.39/23] from ovn-kubernetes
16:07:17	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-f6s8t	Created	Created container: kube-rbac-proxy
16:07:17	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-b59dcb89c-4968k	Started	Started container kube-rbac-proxy
16:07:17	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-b59dcb89c-4968k	Created	Created container: kube-rbac-proxy
16:07:17	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-f6s8t	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:07:17	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-b59dcb89c-4968k	Created	Created container: kserve-container
16:07:17	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-f6s8t	Started	Started container kserve-container
16:07:17	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-f6s8t	Started	Started container kube-rbac-proxy
16:07:17	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-b59dcb89c-4968k	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" in 14.721s (14.721s including waiting). Image size: 1580836240 bytes.
16:07:17	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-f6s8t	Created	Created container: kserve-container
16:07:17	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-b59dcb89c-4968k	Started	Started container kserve-container
16:07:17	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-f6s8t	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" in 15.712s (15.712s including waiting). Image size: 1580836240 bytes.
16:07:17	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-b59dcb89c-4968k	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:07:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
16:07:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94	Created	Created container: kserve-container
16:07:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94	Started	Started container kserve-container
16:07:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:07:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94	Created	Created container: kube-rbac-proxy
16:07:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94	Started	Started container kube-rbac-proxy
16:07:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
16:08:00 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:08:00 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:08:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-v2-runtime] is Ready
16:08:08 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:08:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94	Killing	Stopping container kserve-container
16:08:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94	Killing	Stopping container kube-rbac-proxy
16:08:15	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:08:15	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:08:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94	Unhealthy	Readiness probe failed: Get "http://10.132.0.39:8080/v2/models/isvc-sklearn-v2-runtime/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
16:08:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-68f4f4d797-c6nvr	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-v2-predictor-serving-cert" not found
16:08:21	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-predictor-68f4f4d797 from 0 to 1
16:08:21	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-predictor-68f4f4d797	SuccessfulCreate	Created pod: isvc-sklearn-v2-predictor-68f4f4d797-c6nvr
16:08:21	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
16:08:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-68f4f4d797-c6nvr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:08:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-68f4f4d797-c6nvr	Started	Started container storage-initializer
16:08:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-68f4f4d797-c6nvr	Created	Created container: storage-initializer
16:08:22	kserve-ci-e2e-test	multus	isvc-sklearn-v2-predictor-68f4f4d797-c6nvr	AddedInterface	Add eth0 [10.132.0.40/23] from ovn-kubernetes
16:08:23 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94	Unhealthy	Readiness probe failed: Get "https://10.132.0.39:8643/healthz": dial tcp 10.132.0.39:8643: connect: connection refused
16:08:24 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-b59dcb89c-4968k	Unhealthy	Readiness probe failed: dial tcp 10.133.0.31:8080: connect: connection refused
16:08:24 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-f6s8t	Unhealthy	Readiness probe failed: dial tcp 10.133.0.32:8080: connect: connection refused
16:08:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-68f4f4d797-c6nvr	Started	Started container kube-rbac-proxy
16:08:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-68f4f4d797-c6nvr	Created	Created container: kube-rbac-proxy
16:08:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-68f4f4d797-c6nvr	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:08:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-68f4f4d797-c6nvr	Started	Started container kserve-container
16:08:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-68f4f4d797-c6nvr	Created	Created container: kserve-container
16:08:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-68f4f4d797-c6nvr	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" already present on machine
16:08:34	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-f6s8t	Killing	Stopping container kube-rbac-proxy
16:08:34 (x3)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	UpdateFailed	Failed to update status for InferenceService "isvc-canary-create": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again
16:08:34	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-create-predictor-5674b8c456 from 2 to 1
16:08:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	InferenceServiceReady	InferenceService [isvc-canary-create] is Ready
16:08:34	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-5674b8c456	SuccessfulDelete	Deleted pod: isvc-canary-create-predictor-5674b8c456-f6s8t
16:08:34 (x20)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:08:34	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-f6s8t	Killing	Stopping container kserve-container
16:08:36	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-b59dcb89c-4968k	Killing	Stopping container kube-rbac-proxy
16:08:36	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-b59dcb89c-4968k	Killing	Stopping container kserve-container
16:08:36	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-rd6t5	Killing	Stopping container kube-rbac-proxy
16:08:36	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-rd6t5	Killing	Stopping container kserve-container
16:08:38	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-b59dcb89c-4968k	Unhealthy	Readiness probe failed: Get "https://10.133.0.31:8643/healthz": dial tcp 10.133.0.31:8643: connect: connection refused
16:08:38	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-rd6t5	Unhealthy	Readiness probe failed: Get "https://10.132.0.38:8643/healthz": dial tcp 10.132.0.38:8643: connect: connection refused
16:08:38 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-rd6t5	Unhealthy	Readiness probe failed: dial tcp 10.132.0.38:8080: connect: connection refused
16:08:39	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-5674b8c456-f6s8t	Unhealthy	Readiness probe failed: Get "https://10.133.0.32:8643/healthz": dial tcp 10.133.0.32:8643: connect: connection refused
16:08:41	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-8499c568cb	SuccessfulCreate	Created pod: isvc-canary-promote-predictor-8499c568cb-s5bz7
16:08:41	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-v2-predictor-5654b4fcff	SuccessfulCreate	Created pod: isvc-canary-promote-v2-predictor-5654b4fcff-vws2k
16:08:41	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-v2-predictor-5654b4fcff from 0 to 1
16:08:41	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-8499c568cb	SuccessfulCreate	Created pod: isvc-canary-promote-predictor-8499c568cb-ljpgg
16:08:41	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-predictor-8499c568cb from 0 to 2
16:08:42	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-s5bz7	Started	Started container storage-initializer
16:08:42	kserve-ci-e2e-test	multus	isvc-canary-promote-v2-predictor-5654b4fcff-vws2k	AddedInterface	Add eth0 [10.133.0.33/23] from ovn-kubernetes
16:08:42	kserve-ci-e2e-test	multus	isvc-canary-promote-predictor-8499c568cb-ljpgg	AddedInterface	Add eth0 [10.132.0.41/23] from ovn-kubernetes
16:08:42	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-ljpgg	Started	Started container storage-initializer
16:08:42	kserve-ci-e2e-test	multus	isvc-canary-promote-predictor-8499c568cb-s5bz7	AddedInterface	Add eth0 [10.133.0.34/23] from ovn-kubernetes
16:08:42	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-s5bz7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:08:42	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-ljpgg	Created	Created container: storage-initializer
16:08:42	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-s5bz7	Created	Created container: storage-initializer
16:08:42	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-ljpgg	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:08:42	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-vws2k	Started	Started container storage-initializer
16:08:42	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-vws2k	Created	Created container: storage-initializer
16:08:42	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-vws2k	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:08:46	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-vws2k	Created	Created container: kserve-container
16:08:46	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-s5bz7	Created	Created container: kserve-container
16:08:46	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-vws2k	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:08:46	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-ljpgg	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" already present on machine
16:08:46	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-vws2k	Started	Started container kserve-container
16:08:46	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-ljpgg	Started	Started container kserve-container
16:08:46	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-s5bz7	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" already present on machine
16:08:46	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-vws2k	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" already present on machine
16:08:46	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-s5bz7	Started	Started container kserve-container
16:08:46	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-s5bz7	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:08:46	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-ljpgg	Created	Created container: kube-rbac-proxy
16:08:46	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-ljpgg	Started	Started container kube-rbac-proxy
16:08:46	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-ljpgg	Created	Created container: kserve-container
16:08:46	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-ljpgg	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:08:47	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-s5bz7	Created	Created container: kube-rbac-proxy
16:08:47	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-vws2k	Started	Started container kube-rbac-proxy
16:08:47	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-s5bz7	Started	Started container kube-rbac-proxy
16:08:47	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-vws2k	Created	Created container: kube-rbac-proxy
16:08:51 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:08:51 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:09:40 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-68f4f4d797-c6nvr	Unhealthy	Readiness probe failed: dial tcp 10.132.0.40:8080: connect: connection refused
16:09:50 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:09:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	InferenceServiceReady	InferenceService [isvc-sklearn-v2] is Ready
16:09:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-68f4f4d797-c6nvr	Killing	Stopping container kserve-container
16:09:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-68f4f4d797-c6nvr	Killing	Stopping container kube-rbac-proxy
16:09:51 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:09:51 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:09:54 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-ljpgg	Unhealthy	Readiness probe failed: dial tcp 10.132.0.41:8080: connect: connection refused
16:09:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-68f4f4d797-c6nvr	Unhealthy	Readiness probe failed: Get "https://10.132.0.40:8643/healthz": dial tcp 10.132.0.40:8643: connect: connection refused
16:09:56	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-mixed-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-mixed-predictor-5bb477d776 from 0 to 1
16:09:56	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-mixed-predictor-5bb477d776	SuccessfulCreate	Created pod: isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw
16:09:56	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-mixed": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-mixed": the object has been modified; please apply your changes to the latest version and try again
16:09:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw	Created	Created container: storage-initializer
16:09:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:09:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw	Started	Started container storage-initializer
16:09:57	kserve-ci-e2e-test	multus	isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw	AddedInterface	Add eth0 [10.133.0.35/23] from ovn-kubernetes
16:10:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw	Created	Created container: kube-rbac-proxy
16:10:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw	Started	Started container kube-rbac-proxy
16:10:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:10:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw	Started	Started container kserve-container
16:10:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw	Created	Created container: kserve-container
16:10:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" already present on machine
16:10:04	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-promote-predictor-8499c568cb from 2 to 1
16:10:04	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-ljpgg	Killing	Stopping container kube-rbac-proxy
16:10:04 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again
16:10:04 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	UpdateFailed	Failed to update status for InferenceService "isvc-canary-promote": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again
16:10:04	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-8499c568cb	SuccessfulDelete	Deleted pod: isvc-canary-promote-predictor-8499c568cb-ljpgg
16:10:04	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-ljpgg	Killing	Stopping container kserve-container
16:10:05 (x21)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:10:10	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-ljpgg	Unhealthy	Readiness probe failed: Get "https://10.132.0.41:8643/healthz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
16:10:11	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-v2-predictor-5654b4fcff	SuccessfulCreate	Created pod: isvc-canary-promote-v2-predictor-5654b4fcff-rq467
16:10:11	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-v2-predictor-5654b4fcff from 1 to 2
16:10:11	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-s5bz7	Killing	Stopping container kserve-container
16:10:11	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-s5bz7	Killing	Stopping container kube-rbac-proxy
16:10:12	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-rq467	Started	Started container storage-initializer
16:10:12	kserve-ci-e2e-test	multus	isvc-canary-promote-v2-predictor-5654b4fcff-rq467	AddedInterface	Add eth0 [10.132.0.42/23] from ovn-kubernetes
16:10:12	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-rq467	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:10:12	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-rq467	Created	Created container: storage-initializer
16:10:13 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-s5bz7	Unhealthy	Readiness probe failed: dial tcp 10.133.0.34:8080: connect: connection refused
16:10:13	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-8499c568cb-s5bz7	Unhealthy	Readiness probe failed: Get "https://10.133.0.34:8643/healthz": dial tcp 10.133.0.34:8643: connect: connection refused
16:10:16	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-rq467	Created	Created container: kube-rbac-proxy
16:10:16	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-rq467	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" already present on machine
16:10:16	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-rq467	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:10:16	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-rq467	Started	Started container kube-rbac-proxy
16:10:16	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-rq467	Created	Created container: kserve-container
16:10:16	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-rq467	Started	Started container kserve-container
16:10:41 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:10:41 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:11:19 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:11:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	InferenceServiceReady	InferenceService [isvc-sklearn-v2-mixed] is Ready
16:11:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw	Killing	Stopping container kube-rbac-proxy
16:11:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw	Killing	Stopping container kserve-container
16:11:26 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:11:26 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:11:29 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw	Unhealthy	Readiness probe failed: dial tcp 10.133.0.35:8080: connect: connection refused
16:11:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw	Unhealthy	Readiness probe failed: Get "https://10.133.0.35:8643/healthz": dial tcp 10.133.0.35:8643: connect: connection refused
16:11:31	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-predictor-547dfc5669 from 0 to 1
16:11:31	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-predictor-547dfc5669	SuccessfulCreate	Created pod: isvc-tensorflow-predictor-547dfc5669-9j8q7
16:11:32	kserve-ci-e2e-test	multus	isvc-tensorflow-predictor-547dfc5669-9j8q7	AddedInterface	Add eth0 [10.133.0.36/23] from ovn-kubernetes
16:11:32 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow": the object has been modified; please apply your changes to the latest version and try again
16:11:32	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow": the object has been modified; please apply your changes to the latest version and try again
16:11:32	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-9j8q7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:11:32	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-9j8q7	Created	Created container: storage-initializer
16:11:32	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-9j8q7	Started	Started container storage-initializer
16:11:34 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	InferenceServiceReady	InferenceService [isvc-canary-promote] is Ready
16:11:37	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-9j8q7	Pulling	Pulling image "tensorflow/serving:2.6.2"
16:11:41	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-rq467	Killing	Stopping container kube-rbac-proxy
16:11:41	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-9j8q7	Created	Created container: kube-rbac-proxy
16:11:41	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-9j8q7	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 3.908s (3.908s including waiting). Image size: 425873876 bytes.
16:11:41	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-9j8q7	Created	Created container: kserve-container
16:11:41	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-9j8q7	Started	Started container kserve-container
16:11:41	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-9j8q7	Started	Started container kube-rbac-proxy
16:11:41	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-9j8q7	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:11:41	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-rq467	Killing	Stopping container kserve-container
16:11:41	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-vws2k	Killing	Stopping container kube-rbac-proxy
16:11:41	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-vws2k	Killing	Stopping container kserve-container
16:11:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-vws2k	Unhealthy	Readiness probe failed: Get "https://10.133.0.33:8643/healthz": dial tcp 10.133.0.33:8643: connect: connection refused
16:11:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-rq467	Unhealthy	Readiness probe failed: Get "https://10.132.0.42:8643/healthz": dial tcp 10.132.0.42:8643: connect: connection refused
16:11:44 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-vws2k	Unhealthy	Readiness probe failed: dial tcp 10.133.0.33:8080: connect: connection refused
16:11:44 (x11)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5654b4fcff-rq467	Unhealthy	Readiness probe failed: dial tcp 10.132.0.42:8080: connect: connection refused
16:11:46	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-v2-predictor-fb5f8487c from 0 to 1
16:11:47	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-78f9bdc99b	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-78f9bdc99b-kvk8k
16:11:47	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-kvk8k	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:11:47	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-78f9bdc99b-bgv79	AddedInterface	Add eth0 [10.133.0.38/23] from ovn-kubernetes
16:11:47	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-78f9bdc99b	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-78f9bdc99b-bgv79
16:11:47	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-predictor-78f9bdc99b from 0 to 2
16:11:47	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
16:11:47	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-bgv79	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:11:47	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-bgv79	Created	Created container: storage-initializer
16:11:47	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-bgv79	Started	Started container storage-initializer
16:11:47	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-78f9bdc99b-kvk8k	AddedInterface	Add eth0 [10.132.0.43/23] from ovn-kubernetes
16:11:47	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-v2-predictor-fb5f8487c	SuccessfulCreate	Created pod: isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f
16:11:47	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-kvk8k	Started	Started container storage-initializer
16:11:47	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-kvk8k	Created	Created container: storage-initializer
16:11:48	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f	Started	Started container storage-initializer
16:11:48	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f	Created	Created container: storage-initializer
16:11:48	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:11:48	kserve-ci-e2e-test	multus	isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f	AddedInterface	Add eth0 [10.133.0.37/23] from ovn-kubernetes
16:11:49 (x4)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-9j8q7	Unhealthy	Readiness probe failed: dial tcp 10.133.0.36:8080: connect: connection refused
16:11:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-bgv79	Created	Created container: kserve-container
16:11:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-bgv79	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:11:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-bgv79	Started	Started container kube-rbac-proxy
16:11:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f	Started	Started container kube-rbac-proxy
16:11:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f	Created	Created container: kube-rbac-proxy
16:11:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:11:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-bgv79	Created	Created container: kube-rbac-proxy
16:11:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-kvk8k	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" already present on machine
16:11:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f	Started	Started container kserve-container
16:11:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-kvk8k	Started	Started container kube-rbac-proxy
16:11:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-bgv79	Started	Started container kserve-container
16:11:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f	Created	Created container: kserve-container
16:11:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" already present on machine
16:11:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-kvk8k	Created	Created container: kserve-container
16:11:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-bgv79	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" already present on machine
16:11:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-kvk8k	Started	Started container kserve-container
16:11:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-kvk8k	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:11:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-kvk8k	Created	Created container: kube-rbac-proxy
16:11:59 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:11:59	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	InferenceServiceReady	InferenceService [isvc-tensorflow] is Ready
16:12:01 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:12:01 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:12:14	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-9j8q7	Killing	Stopping container kube-rbac-proxy
16:12:14	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-9j8q7	Killing	Stopping container kserve-container
16:12:19	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-runtime-predictor-855cfcd677	SuccessfulCreate	Created pod: isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv
16:12:19	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-runtime-predictor-855cfcd677 from 0 to 1
16:12:19 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow-runtime": the object has been modified; please apply your changes to the latest version and try again
16:12:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow-runtime": the object has been modified; please apply your changes to the latest version and try again
16:12:20	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv	Started	Started container storage-initializer
16:12:20	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:12:20	kserve-ci-e2e-test	multus	isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv	AddedInterface	Add eth0 [10.132.0.44/23] from ovn-kubernetes
16:12:20	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv	Created	Created container: storage-initializer
16:12:25	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv	Pulling	Pulling image "tensorflow/serving:2.6.2"
16:12:29	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv	Created	Created container: kube-rbac-proxy
16:12:29	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:12:29	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 3.591s (3.591s including waiting). Image size: 425873876 bytes.
16:12:29	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv	Started	Started container kube-rbac-proxy
16:12:29	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv	Created	Created container: kserve-container
16:12:29	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv	Started	Started container kserve-container
16:12:34	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:12:34	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:12:35 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv	Unhealthy	Readiness probe failed: dial tcp 10.132.0.44:8080: connect: connection refused
16:12:44 (x7)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-9j8q7	Unhealthy	Readiness probe failed: Get "https://10.133.0.36:8643/healthz": dial tcp 10.133.0.36:8643: connect: connection refused
16:12:45	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	InferenceServiceReady	InferenceService [isvc-tensorflow-runtime] is Ready
16:12:45 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:12:49	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:12:49	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:12:59 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-bgv79	Unhealthy	Readiness probe failed: dial tcp 10.133.0.38:8080: connect: connection refused
16:13:01	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv	Killing	Stopping container kserve-container
16:13:01	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv	Killing	Stopping container kube-rbac-proxy
16:13:04 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-kvk8k	Unhealthy	Readiness probe failed: dial tcp 10.132.0.43:8080: connect: connection refused
16:13:06	kserve-ci-e2e-test	deployment-controller	isvc-triton-predictor	ScalingReplicaSet	Scaled up replica set isvc-triton-predictor-5d7db58676 from 0 to 1
16:13:06	kserve-ci-e2e-test	multus	isvc-triton-predictor-5d7db58676-zwxtg	AddedInterface	Add eth0 [10.133.0.39/23] from ovn-kubernetes
16:13:06	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zwxtg	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:13:06	kserve-ci-e2e-test	replicaset-controller	isvc-triton-predictor-5d7db58676	SuccessfulCreate	Created pod: isvc-triton-predictor-5d7db58676-zwxtg
16:13:06	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	UpdateFailed	Failed to update status for InferenceService "isvc-triton": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-triton": the object has been modified; please apply your changes to the latest version and try again
16:13:07	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-bgv79	Killing	Stopping container kube-rbac-proxy
16:13:07	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-rollback-predictor-78f9bdc99b from 2 to 1
16:13:07 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again
16:13:07 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	UpdateFailed	Failed to update status for InferenceService "isvc-canary-rollback": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again
16:13:07	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zwxtg	Started	Started container storage-initializer
16:13:07	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zwxtg	Created	Created container: storage-initializer
16:13:07	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-bgv79	Killing	Stopping container kserve-container
16:13:07	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-78f9bdc99b	SuccessfulDelete	Deleted pod: isvc-canary-rollback-predictor-78f9bdc99b-bgv79
16:13:09	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-bgv79	Unhealthy	Readiness probe failed: Get "https://10.133.0.38:8643/healthz": dial tcp 10.133.0.38:8643: connect: connection refused
16:13:12	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zwxtg	Pulling	Pulling image "nvcr.io/nvidia/tritonserver:23.05-py3"
16:13:16	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f	Killing	Stopping container kserve-container
16:13:16	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-78f9bdc99b	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-78f9bdc99b-9vph6
16:13:16	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f	Killing	Stopping container kube-rbac-proxy
16:13:16	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-predictor-78f9bdc99b from 1 to 2
16:13:16 (x21)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:13:17 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f	Unhealthy	Readiness probe failed: dial tcp 10.133.0.37:8080: connect: connection refused
16:13:19	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f	Unhealthy	Readiness probe failed: Get "https://10.133.0.37:8643/healthz": dial tcp 10.133.0.37:8643: connect: connection refused
16:13:19	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-78f9bdc99b-9vph6	AddedInterface	Add eth0 [10.133.0.40/23] from ovn-kubernetes
16:13:22	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-9vph6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:13:22	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-9vph6	Started	Started container storage-initializer
16:13:22	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-9vph6	Created	Created container: storage-initializer
16:13:28	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-9vph6	Created	Created container: kube-rbac-proxy
16:13:28	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-9vph6	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" already present on machine
16:13:28	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-9vph6	Started	Started container kserve-container
16:13:28	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-9vph6	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:13:28	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-9vph6	Started	Started container kube-rbac-proxy
16:13:28	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-9vph6	Created	Created container: kserve-container
16:13:30 (x6)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv	Unhealthy	Readiness probe failed: Get "https://10.132.0.44:8643/healthz": dial tcp 10.132.0.44:8643: connect: connection refused
16:15:01 (x12)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-9vph6	Unhealthy	Readiness probe failed: dial tcp 10.133.0.40:8080: connect: connection refused
16:15:06 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:15:06 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:15:07	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zwxtg	Pulled	Successfully pulled image "nvcr.io/nvidia/tritonserver:23.05-py3" in 1m55.835s (1m55.835s including waiting). Image size: 12907074623 bytes.
16:15:07	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zwxtg	Created	Created container: kserve-container
16:15:08	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zwxtg	Created	Created container: kube-rbac-proxy
16:15:08	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zwxtg	Started	Started container kube-rbac-proxy
16:15:08	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zwxtg	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:15:08	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zwxtg	Started	Started container kserve-container
16:15:09 (x2)	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zwxtg	Unhealthy	Readiness probe failed: dial tcp 10.133.0.39:8080: connect: connection refused
16:15:11 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	InferenceServiceReady	InferenceService [isvc-canary-rollback] is Ready
16:15:14 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:15:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	InferenceServiceReady	InferenceService [isvc-triton] is Ready
16:15:17	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-kvk8k	Killing	Stopping container kserve-container
16:15:17	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-kvk8k	Killing	Stopping container kube-rbac-proxy
16:15:17	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-9vph6	Killing	Stopping container kserve-container
16:15:17	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-9vph6	Killing	Stopping container kube-rbac-proxy
16:15:18	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zwxtg	Killing	Stopping container kube-rbac-proxy
16:15:18	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zwxtg	Killing	Stopping container kserve-container
16:15:19	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zwxtg	Unhealthy	Readiness probe failed: Get "https://10.133.0.39:8643/healthz": dial tcp 10.133.0.39:8643: connect: connection refused
16:15:19	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-kvk8k	Unhealthy	Readiness probe failed: Get "https://10.132.0.43:8643/healthz": dial tcp 10.132.0.43:8643: connect: connection refused
16:15:20	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-9vph6	Unhealthy	Readiness probe failed: Get "https://10.133.0.40:8643/healthz": dial tcp 10.133.0.40:8643: connect: connection refused
16:15:22	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-556d599d48	SuccessfulCreate	Created pod: isvc-canary-stop-predictor-556d599d48-nhvnp
16:15:22	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
16:15:22	kserve-ci-e2e-test	multus	isvc-canary-stop-predictor-556d599d48-nhvnp	AddedInterface	Add eth0 [10.132.0.45/23] from ovn-kubernetes
16:15:22	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-v2-predictor-5cbd69c75c	SuccessfulCreate	Created pod: isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l
16:15:22	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-stop-v2-predictor-5cbd69c75c from 0 to 1
16:15:22	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-nhvnp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:15:22	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-556d599d48	SuccessfulCreate	Created pod: isvc-canary-stop-predictor-556d599d48-pcrdx
16:15:22	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-78f9bdc99b-9vph6	Unhealthy	Readiness probe failed: dial tcp 10.133.0.40:8080: i/o timeout
16:15:22	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-stop-predictor-556d599d48 from 0 to 2
16:15:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-nhvnp	Created	Created container: storage-initializer
16:15:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-nhvnp	Started	Started container storage-initializer
16:15:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:15:23	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-predictor-7566d65bf7	SuccessfulCreate	Created pod: isvc-xgboost-predictor-7566d65bf7-q4ck9
16:15:23	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-predictor-7566d65bf7 from 0 to 1
16:15:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-pcrdx	Started	Started container storage-initializer
16:15:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-pcrdx	Created	Created container: storage-initializer
16:15:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-pcrdx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:15:23	kserve-ci-e2e-test	multus	isvc-canary-stop-predictor-556d599d48-pcrdx	AddedInterface	Add eth0 [10.133.0.42/23] from ovn-kubernetes
16:15:23	kserve-ci-e2e-test	multus	isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l	AddedInterface	Add eth0 [10.133.0.41/23] from ovn-kubernetes
16:15:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l	Started	Started container storage-initializer
16:15:23	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost": the object has been modified; please apply your changes to the latest version and try again
16:15:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l	Created	Created container: storage-initializer
16:15:24	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-q4ck9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:15:24	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-q4ck9	Started	Started container storage-initializer
16:15:24	kserve-ci-e2e-test	multus	isvc-xgboost-predictor-7566d65bf7-q4ck9	AddedInterface	Add eth0 [10.132.0.46/23] from ovn-kubernetes
16:15:24	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-q4ck9	Created	Created container: storage-initializer
16:15:27	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l	Created	Created container: kube-rbac-proxy
16:15:27	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-pcrdx	Created	Created container: kserve-container
16:15:27	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-nhvnp	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" already present on machine
16:15:27	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-nhvnp	Created	Created container: kserve-container
16:15:27	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-nhvnp	Started	Started container kserve-container
16:15:27	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-nhvnp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:15:27	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-nhvnp	Created	Created container: kube-rbac-proxy
16:15:27	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-nhvnp	Started	Started container kube-rbac-proxy
16:15:27	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-pcrdx	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" already present on machine
16:15:27	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-pcrdx	Started	Started container kserve-container
16:15:27	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l	Started	Started container kube-rbac-proxy
16:15:27	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:15:27	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l	Started	Started container kserve-container
16:15:27	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l	Created	Created container: kserve-container
16:15:27	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" already present on machine
16:15:27	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-pcrdx	Started	Started container kube-rbac-proxy
16:15:27	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-pcrdx	Created	Created container: kube-rbac-proxy
16:15:27	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-pcrdx	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:15:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-q4ck9	Pulling	Pulling image "kserve/xgbserver:latest"
16:15:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-q4ck9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:15:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-q4ck9	Started	Started container kube-rbac-proxy
16:15:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-q4ck9	Created	Created container: kube-rbac-proxy
16:15:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-q4ck9	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 20.545s (20.545s including waiting). Image size: 1435742979 bytes.
16:15:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-q4ck9	Created	Created container: kserve-container
16:15:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-q4ck9	Started	Started container kserve-container
16:16:23 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:16:23 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:16:35 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-nhvnp	Unhealthy	Readiness probe failed: dial tcp 10.132.0.45:8080: connect: connection refused
16:16:44	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-556d599d48	SuccessfulDelete	Deleted pod: isvc-canary-stop-predictor-556d599d48-nhvnp
16:16:44 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	UpdateFailed	Failed to update status for InferenceService "isvc-canary-stop": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again
16:16:44 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again
16:16:44	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-nhvnp	Killing	Stopping container kube-rbac-proxy
16:16:44	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-nhvnp	Killing	Stopping container kserve-container
16:16:44	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-stop-predictor-556d599d48 from 2 to 1
16:16:45	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	InferenceServiceReady	InferenceService [isvc-canary-stop] is Ready
16:16:45	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-nhvnp	Unhealthy	Readiness probe failed: Get "https://10.132.0.45:8643/healthz": dial tcp 10.132.0.45:8643: connect: connection refused
16:16:52	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l	Killing	Stopping container kserve-container
16:16:52 (x20)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:16:52	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	InferenceServiceNotReady	InferenceService [isvc-canary-stop] is no longer Ready because of: PredictorReady, Ready
16:16:52	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-pcrdx	Killing	Stopping container kube-rbac-proxy
16:16:52	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l	Killing	Stopping container kube-rbac-proxy
16:16:52	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-pcrdx	Killing	Stopping container kserve-container
16:16:53 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:16:53 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:16:54	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-pcrdx	Unhealthy	Readiness probe failed: Get "https://10.133.0.42:8643/healthz": dial tcp 10.133.0.42:8643: connect: connection refused
16:16:54 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l	Unhealthy	Readiness probe failed: dial tcp 10.133.0.41:8080: connect: connection refused
16:16:54	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l	Unhealthy	Readiness probe failed: Get "https://10.133.0.41:8643/healthz": dial tcp 10.133.0.41:8643: connect: connection refused
16:16:54 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-556d599d48-pcrdx	Unhealthy	Readiness probe failed: dial tcp 10.133.0.42:8080: connect: connection refused
16:16:56 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:16:56	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	InferenceServiceReady	InferenceService [isvc-xgboost] is Ready
16:17:02	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm": the object has been modified; please apply your changes to the latest version and try again
16:17:02	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-predictor-54f85ddf88	SuccessfulCreate	Created pod: isvc-lightgbm-predictor-54f85ddf88-rl5dc
16:17:02	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-predictor-54f85ddf88 from 0 to 1
16:17:02 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm": the object has been modified; please apply your changes to the latest version and try again
16:17:03	kserve-ci-e2e-test	multus	isvc-lightgbm-predictor-54f85ddf88-rl5dc	AddedInterface	Add eth0 [10.133.0.43/23] from ovn-kubernetes
16:17:03	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-rl5dc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:17:03	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-rl5dc	Created	Created container: storage-initializer
16:17:03	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-rl5dc	Started	Started container storage-initializer
16:17:04	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-q4ck9	Killing	Stopping container kserve-container
16:17:04	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-q4ck9	Killing	Stopping container kube-rbac-proxy
16:17:06 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-q4ck9	Unhealthy	Readiness probe failed: dial tcp 10.132.0.46:8080: connect: connection refused
16:17:06	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-q4ck9	Unhealthy	Readiness probe failed: Get "https://10.132.0.46:8643/healthz": dial tcp 10.132.0.46:8643: connect: connection refused
16:17:08	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-rl5dc	Pulling	Pulling image "kserve/lgbserver:latest"
16:17:09	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
16:17:09	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-mlserver-predictor-7b55b84f54 from 0 to 1
16:17:09	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-mlserver-predictor-7b55b84f54	SuccessfulCreate	Created pod: isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9
16:17:10	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:17:10	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9	Started	Started container storage-initializer
16:17:10	kserve-ci-e2e-test	multus	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9	AddedInterface	Add eth0 [10.132.0.47/23] from ovn-kubernetes
16:17:10	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9	Created	Created container: storage-initializer
16:17:14	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9	Created	Created container: kserve-container
16:17:14	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9	Started	Started container kube-rbac-proxy
16:17:14	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9	Started	Started container kserve-container
16:17:14	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
16:17:14	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9	Created	Created container: kube-rbac-proxy
16:17:14	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:17:15	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-rl5dc	Started	Started container kube-rbac-proxy
16:17:15	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-rl5dc	Created	Created container: kube-rbac-proxy
16:17:15	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-rl5dc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:17:15	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-rl5dc	Started	Started container kserve-container
16:17:15	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-rl5dc	Created	Created container: kserve-container
16:17:15	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-rl5dc	Pulled	Successfully pulled image "kserve/lgbserver:latest" in 6.858s (6.858s including waiting). Image size: 643927316 bytes.
16:17:39	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:17:39	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:17:52	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	InferenceServiceReady	InferenceService [isvc-xgboost-v2-mlserver] is Ready
16:17:52 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:17:54 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:17:54 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:17:59	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9	Killing	Stopping container kube-rbac-proxy
16:17:59	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9	Killing	Stopping container kserve-container
16:18:02	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9	Unhealthy	Readiness probe failed: Get "https://10.132.0.47:8643/healthz": dial tcp 10.132.0.47:8643: connect: connection refused
16:18:02	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9	Unhealthy	Readiness probe failed: Get "http://10.132.0.47:8080/v2/models/isvc-xgboost-v2-mlserver/ready": dial tcp 10.132.0.47:8080: connect: connection refused
16:18:02 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:18:02 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:18:04 (x2)	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
16:18:04	kserve-ci-e2e-test	deployment-controller	xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set xgboost-v2-mlserver-predictor-59944bcf4c from 0 to 1
16:18:04	kserve-ci-e2e-test	replicaset-controller	xgboost-v2-mlserver-predictor-59944bcf4c	SuccessfulCreate	Created pod: xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw
16:18:04	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
16:18:05	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "xgboost-v2-mlserver-predictor-serving-cert" not found
16:18:06	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw	Created	Created container: storage-initializer
16:18:06	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw	Started	Started container storage-initializer
16:18:06	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:18:06	kserve-ci-e2e-test	multus	xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw	AddedInterface	Add eth0 [10.132.0.48/23] from ovn-kubernetes
16:18:10	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:18:10	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw	Started	Started container kserve-container
16:18:10	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw	Created	Created container: kserve-container
16:18:10	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
16:18:11	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw	Created	Created container: kube-rbac-proxy
16:18:11	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw	Started	Started container kube-rbac-proxy
16:18:19	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:18:19	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:18:42 (x12)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-rl5dc	Unhealthy	Readiness probe failed: dial tcp 10.133.0.43:8080: connect: connection refused
16:18:49	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	InferenceServiceReady	InferenceService [xgboost-v2-mlserver] is Ready
16:18:49 (x11)	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:18:49 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:18:49 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:18:52	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	InferenceServiceReady	InferenceService [isvc-lightgbm] is Ready
16:18:52 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:18:55	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw	Killing	Stopping container kube-rbac-proxy
16:18:55	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw	Killing	Stopping container kserve-container
16:18:58	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw	Unhealthy	Readiness probe failed: Get "http://10.132.0.48:8080/v2/models/xgboost-v2-mlserver/ready": dial tcp 10.132.0.48:8080: connect: connection refused
16:18:58	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw	Unhealthy	Readiness probe failed: Get "https://10.132.0.48:8643/healthz": dial tcp 10.132.0.48:8643: connect: connection refused
16:19:00	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-tl8h4	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
16:19:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-runtime": the object has been modified; please apply your changes to the latest version and try again
16:19:00	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-runtime-predictor-54dd488c85 from 0 to 1
16:19:00	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-runtime-predictor-54dd488c85	SuccessfulCreate	Created pod: isvc-xgboost-runtime-predictor-54dd488c85-tl8h4
16:19:01	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-tl8h4	Started	Started container storage-initializer
16:19:01	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-tl8h4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:19:01	kserve-ci-e2e-test	multus	isvc-xgboost-runtime-predictor-54dd488c85-tl8h4	AddedInterface	Add eth0 [10.133.0.44/23] from ovn-kubernetes
16:19:01	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-tl8h4	Created	Created container: storage-initializer
16:19:02 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:19:02 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:19:03	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-rl5dc	Killing	Stopping container kube-rbac-proxy
16:19:03	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-rl5dc	Killing	Stopping container kserve-container
16:19:05	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-tl8h4	Pulling	Pulling image "kserve/xgbserver:latest"
16:19:07	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-rl5dc	Unhealthy	Readiness probe failed: Get "https://10.133.0.43:8643/healthz": dial tcp 10.133.0.43:8643: connect: connection refused
16:19:08	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-runtime-predictor-577f7db7cb	SuccessfulCreate	Created pod: isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9
16:19:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-runtime": the object has been modified; please apply your changes to the latest version and try again
16:19:08	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-runtime-predictor-577f7db7cb from 0 to 1
16:19:09	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9	Created	Created container: storage-initializer
16:19:09	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9	Started	Started container storage-initializer
16:19:09	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:19:09	kserve-ci-e2e-test	multus	isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9	AddedInterface	Add eth0 [10.132.0.49/23] from ovn-kubernetes
16:19:14	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9	Pulling	Pulling image "kserve/lgbserver:latest"
16:19:20	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9	Created	Created container: kube-rbac-proxy
16:19:20	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:19:20	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9	Started	Started container kserve-container
16:19:20	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9	Created	Created container: kserve-container
16:19:20	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9	Pulled	Successfully pulled image "kserve/lgbserver:latest" in 6.142s (6.142s including waiting). Image size: 643927316 bytes.
16:19:20	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9	Started	Started container kube-rbac-proxy
16:19:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-tl8h4	Created	Created container: kserve-container
16:19:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-tl8h4	Started	Started container kube-rbac-proxy
16:19:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-tl8h4	Created	Created container: kube-rbac-proxy
16:19:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-tl8h4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:19:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-tl8h4	Started	Started container kserve-container
16:19:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-tl8h4	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 20.557s (20.557s including waiting). Image size: 1435742979 bytes.
16:19:53 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:19:53 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:20:00 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:20:00 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:20:42 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:20:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-runtime] is Ready
16:20:45 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:20:45 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:20:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-runtime] is Ready
16:20:50 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:20:50	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-tl8h4	Killing	Stopping container kserve-container
16:20:50	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-tl8h4	Killing	Stopping container kube-rbac-proxy
16:20:52	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-tl8h4	Unhealthy	Readiness probe failed: Get "https://10.133.0.44:8643/healthz": dial tcp 10.133.0.44:8643: connect: connection refused
16:20:52 (x10)	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-tl8h4	Unhealthy	Readiness probe failed: dial tcp 10.133.0.44:8080: connect: connection refused
16:20:53 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:20:53 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:20:55	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-runtime-predictor-7787c55dc9 from 0 to 1
16:20:56 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
16:20:56	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-runtime-predictor-7787c55dc9	SuccessfulCreate	Created pod: isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk
16:20:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk	Started	Started container storage-initializer
16:20:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk	Created	Created container: storage-initializer
16:20:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:20:56	kserve-ci-e2e-test	multus	isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk	AddedInterface	Add eth0 [10.132.0.50/23] from ovn-kubernetes
16:20:56	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
16:21:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9	Killing	Stopping container kube-rbac-proxy
16:21:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9	Killing	Stopping container kserve-container
16:21:00 (x12)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9	Unhealthy	Readiness probe failed: dial tcp 10.132.0.49:8080: connect: connection refused
16:21:01	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:21:01	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk	Created	Created container: kube-rbac-proxy
16:21:01	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk	Started	Started container kube-rbac-proxy
16:21:01	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk	Created	Created container: kserve-container
16:21:01	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk	Started	Started container kserve-container
16:21:01	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
16:21:03	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9	Unhealthy	Readiness probe failed: Get "https://10.132.0.49:8643/healthz": dial tcp 10.132.0.49:8643: connect: connection refused
16:21:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w	Created	Created container: storage-initializer
16:21:05	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w	AddedInterface	Add eth0 [10.132.0.51/23] from ovn-kubernetes
16:21:05	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
16:21:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:21:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w	Started	Started container storage-initializer
16:21:05	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-runtime-predictor-7f4957c5d5 from 0 to 1
16:21:05	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5	SuccessfulCreate	Created pod: isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w
16:21:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:21:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
16:21:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w	Created	Created container: kserve-container
16:21:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w	Started	Started container kserve-container
16:21:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w	Started	Started container kube-rbac-proxy
16:21:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w	Created	Created container: kube-rbac-proxy
16:21:25	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:21:26	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:21:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
16:21:41 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:21:41 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:21:48 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:21:48	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-runtime] is Ready
16:21:49 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:21:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-v2-runtime] is Ready
16:21:50 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:21:50 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:21:55	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w	Killing	Stopping container kserve-container
16:21:55	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w	Killing	Stopping container kube-rbac-proxy
16:21:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk	Killing	Stopping container kserve-container
16:21:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk	Killing	Stopping container kube-rbac-proxy
16:21:59	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk	Unhealthy	Readiness probe failed: Get "http://10.132.0.50:8080/v2/models/isvc-xgboost-v2-runtime/ready": dial tcp 10.132.0.50:8080: connect: connection refused
16:21:59	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk	Unhealthy	Readiness probe failed: Get "https://10.132.0.50:8643/healthz": dial tcp 10.132.0.50:8643: connect: connection refused
16:22:00	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5 from 0 to 1
16:22:00	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5	SuccessfulCreate	Created pod: isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx
16:22:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
16:22:01	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx	AddedInterface	Add eth0 [10.133.0.45/23] from ovn-kubernetes
16:22:01	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:22:01	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx	Created	Created container: storage-initializer
16:22:01	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-predictor-fcc668bc from 0 to 1
16:22:01	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-tdbqd	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-v2-predictor-serving-cert" not found
16:22:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
16:22:01	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-predictor-fcc668bc	SuccessfulCreate	Created pod: isvc-xgboost-v2-predictor-fcc668bc-tdbqd
16:22:01	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx	Started	Started container storage-initializer
16:22:02	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-tdbqd	Created	Created container: storage-initializer
16:22:02	kserve-ci-e2e-test	multus	isvc-xgboost-v2-predictor-fcc668bc-tdbqd	AddedInterface	Add eth0 [10.132.0.52/23] from ovn-kubernetes
16:22:02	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-tdbqd	Started	Started container storage-initializer
16:22:02	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-tdbqd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:22:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx	Started	Started container kserve-container
16:22:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx	Created	Created container: kserve-container
16:22:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx	Created	Created container: kube-rbac-proxy
16:22:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx	Started	Started container kube-rbac-proxy
16:22:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx	Pulled	Container image "kserve/lgbserver:latest" already present on machine
16:22:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:22:06	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-tdbqd	Pulled	Container image "kserve/xgbserver:latest" already present on machine
16:22:07	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-tdbqd	Started	Started container kserve-container
16:22:07	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-tdbqd	Started	Started container kube-rbac-proxy
16:22:07	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-tdbqd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:22:07	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-tdbqd	Created	Created container: kube-rbac-proxy
16:22:07	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-tdbqd	Created	Created container: kserve-container
16:22:12 (x3)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx	Unhealthy	Readiness probe failed: dial tcp 10.133.0.45:8080: connect: connection refused
16:22:15	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:22:15	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:22:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-kserve] is Ready
16:22:22 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:22:30	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:22:30	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx	Killing	Stopping container kserve-container
16:22:30	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx	Killing	Stopping container kube-rbac-proxy
16:22:30	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:22:35	kserve-ci-e2e-test	replicaset-controller	isvc-mlflow-v2-runtime-predictor-6456f94494	SuccessfulCreate	Created pod: isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2
16:22:35	kserve-ci-e2e-test	deployment-controller	isvc-mlflow-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-mlflow-v2-runtime-predictor-6456f94494 from 0 to 1
16:22:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-mlflow-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-mlflow-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
16:22:36	kserve-ci-e2e-test	multus	isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2	AddedInterface	Add eth0 [10.132.0.53/23] from ovn-kubernetes
16:22:36	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
16:22:37	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:22:37	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2	Created	Created container: storage-initializer
16:22:37	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2	Started	Started container storage-initializer
16:22:41	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2	Started	Started container kube-rbac-proxy
16:22:41	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2	Created	Created container: kube-rbac-proxy
16:22:41	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
16:22:41	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2	Created	Created container: kserve-container
16:22:41	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2	Started	Started container kserve-container
16:22:41	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:22:46 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:22:46 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:23:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	InferenceServiceReady	InferenceService [isvc-xgboost-v2] is Ready
16:23:15 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:23:16 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:23:16 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:23:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	InferenceServiceReady	InferenceService [isvc-mlflow-v2-runtime] is Ready
16:23:19 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:23:20 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:23:20 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:23:21	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-tdbqd	Killing	Stopping container kube-rbac-proxy
16:23:21	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-tdbqd	Killing	Stopping container kserve-container
16:23:24 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-tdbqd	Unhealthy	Readiness probe failed: dial tcp 10.132.0.52:8080: connect: connection refused
16:23:24	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-tdbqd	Unhealthy	Readiness probe failed: Get "https://10.132.0.52:8643/healthz": dial tcp 10.132.0.52:8643: connect: connection refused
16:23:25	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2	Killing	Stopping container kserve-container
16:23:25	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2	Killing	Stopping container kube-rbac-proxy
16:23:27	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-predictor-547bfff996 from 0 to 1
16:23:27	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3": the object has been modified; please apply your changes to the latest version and try again
16:23:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-547bfff996-2r8wd	Created	Created container: storage-initializer
16:23:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-547bfff996-2r8wd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:23:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-547bfff996-2r8wd	Started	Started container storage-initializer
16:23:27	kserve-ci-e2e-test	multus	isvc-sklearn-s3-predictor-547bfff996-2r8wd	AddedInterface	Add eth0 [10.133.0.46/23] from ovn-kubernetes
16:23:27	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-predictor-547bfff996	SuccessfulCreate	Created pod: isvc-sklearn-s3-predictor-547bfff996-2r8wd
16:23:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-547bfff996-2r8wd	Started	Started container kube-rbac-proxy
16:23:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-547bfff996-2r8wd	Created	Created container: kube-rbac-proxy
16:23:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-547bfff996-2r8wd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:23:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-547bfff996-2r8wd	Created	Created container: kserve-container
16:23:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-547bfff996-2r8wd	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" already present on machine
16:23:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-547bfff996-2r8wd	Started	Started container kserve-container
16:23:31	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-mcp-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-mcp-predictor-6c6bf9d8c5 from 0 to 1
16:23:31	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-mcp-predictor-6c6bf9d8c5	SuccessfulCreate	Created pod: isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj
16:23:31	kserve-ci-e2e-test	multus	isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj	AddedInterface	Add eth0 [10.133.0.47/23] from ovn-kubernetes
16:23:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:23:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj	Created	Created container: storage-initializer
16:23:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-mcp": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-mcp": the object has been modified; please apply your changes to the latest version and try again
16:23:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj	Started	Started container storage-initializer
16:23:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj	Pulling	Pulling image "quay.io/opendatahub/kserve-agent:latest"
16:23:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj	Started	Started container kserve-container
16:23:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" already present on machine
16:23:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj	Created	Created container: kserve-container
16:23:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj	Created	Created container: kube-rbac-proxy
16:23:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj	Started	Started container kserve-agent
16:23:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent:latest" in 5.183s (5.183s including waiting). Image size: 239481897 bytes.
16:23:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj	Created	Created container: kserve-agent
16:23:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:23:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj	Started	Started container kube-rbac-proxy
16:23:57 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:23:57 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:24:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj	Unhealthy	Liveness probe failed: Get "http://10.133.0.47:8080/v1/models/isvc-sklearn-mcp": dial tcp 10.133.0.47:8080: connect: connection refused
16:24:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	InferenceServiceReady	InferenceService [isvc-sklearn-s3] is Ready
16:24:47 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:24:48 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:24:48	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	InferenceServiceReady	InferenceService [isvc-sklearn-mcp] is Ready
16:24:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj	Killing	Stopping container kube-rbac-proxy
16:24:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj	Killing	Stopping container kserve-container
16:24:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj	Killing	Stopping container kserve-agent
16:24:56	kserve-ci-e2e-test	deployment-controller	isvc-paddle-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-predictor-6d5c9fd4f6 from 0 to 1
16:24:56	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-pzrq5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:24:56	kserve-ci-e2e-test	multus	isvc-paddle-predictor-6d5c9fd4f6-pzrq5	AddedInterface	Add eth0 [10.132.0.54/23] from ovn-kubernetes
16:24:56	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	UpdateFailed	Failed to update status for InferenceService "isvc-paddle": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle": the object has been modified; please apply your changes to the latest version and try again
16:24:56	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-predictor-6d5c9fd4f6	SuccessfulCreate	Created pod: isvc-paddle-predictor-6d5c9fd4f6-pzrq5
16:24:57 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:24:57 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:24:57	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-pzrq5	Created	Created container: storage-initializer
16:24:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-547bfff996-2r8wd	Killing	Stopping container kserve-container
16:24:57	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-pass-predictor-676f88c84	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8
16:24:57	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-pass-predictor-676f88c84 from 0 to 1
16:24:57	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-pass": the object has been modified; please apply your changes to the latest version and try again
16:24:57 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-547bfff996-2r8wd	Unhealthy	Readiness probe failed: dial tcp 10.133.0.46:8080: connect: connection refused
16:24:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
16:24:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-547bfff996-2r8wd	Killing	Stopping container kube-rbac-proxy
16:24:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-547bfff996-2r8wd	Unhealthy	Readiness probe failed: Get "https://10.133.0.46:8643/healthz": dial tcp 10.133.0.46:8643: connect: connection refused
16:24:57	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-pzrq5	Started	Started container storage-initializer
16:24:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:24:58	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8	AddedInterface	Add eth0 [10.132.0.55/23] from ovn-kubernetes
16:24:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8	Started	Started container storage-initializer
16:24:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8	Created	Created container: storage-initializer
16:25:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8	Created	Created container: kserve-container
16:25:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" already present on machine
16:25:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8	Started	Started container kube-rbac-proxy
16:25:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8	Created	Created container: kube-rbac-proxy
16:25:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:25:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8	Started	Started container kserve-container
16:25:01	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-pzrq5	Pulling	Pulling image "kserve/paddleserver:latest"
16:25:13	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-pzrq5	Started	Started container kserve-container
16:25:13	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-pzrq5	Started	Started container kube-rbac-proxy
16:25:13	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-pzrq5	Created	Created container: kube-rbac-proxy
16:25:13	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-pzrq5	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:25:13	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-pzrq5	Pulled	Successfully pulled image "kserve/paddleserver:latest" in 11.959s (11.959s including waiting). Image size: 1203659522 bytes.
16:25:13	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-pzrq5	Created	Created container: kserve-container
16:25:18 (x4)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj	Unhealthy	Readiness probe failed: Get "http://10.133.0.47:8080/v1/models/isvc-sklearn-mcp": dial tcp 10.133.0.47:8080: connect: connection refused
16:25:18 (x6)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj	Unhealthy	Readiness probe failed: Get "https://10.133.0.47:8643/healthz": dial tcp 10.133.0.47:8643: connect: connection refused
16:25:27 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:25:27 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:25:50 (x6)	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-pzrq5	Unhealthy	Readiness probe failed: dial tcp 10.132.0.54:8080: connect: connection refused
16:25:56 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:25:56 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:26:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	InferenceServiceReady	InferenceService [isvc-paddle] is Ready
16:26:01 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:26:07	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-pzrq5	Killing	Stopping container kube-rbac-proxy
16:26:07	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-pzrq5	Killing	Stopping container kserve-container
16:26:12 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:26:12 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:26:12	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-runtime": the object has been modified; please apply your changes to the latest version and try again
16:26:12	kserve-ci-e2e-test	deployment-controller	isvc-paddle-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-runtime-predictor-58b99fff76 from 0 to 1
16:26:13	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-runtime-predictor-58b99fff76	SuccessfulCreate	Created pod: isvc-paddle-runtime-predictor-58b99fff76-kt7kv
16:26:13	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-kt7kv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:26:13	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-kt7kv	Created	Created container: storage-initializer
16:26:13	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-kt7kv	Started	Started container storage-initializer
16:26:13	kserve-ci-e2e-test	multus	isvc-paddle-runtime-predictor-58b99fff76-kt7kv	AddedInterface	Add eth0 [10.133.0.48/23] from ovn-kubernetes
16:26:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-global-pass] is Ready
16:26:18 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:26:19	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-kt7kv	Pulling	Pulling image "kserve/paddleserver:latest"
16:26:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8	Killing	Stopping container kserve-container
16:26:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8	Killing	Stopping container kube-rbac-proxy
16:26:28 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-fail": the object has been modified; please apply your changes to the latest version and try again
16:26:28 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8	Unhealthy	Readiness probe failed: dial tcp 10.132.0.55:8080: connect: connection refused
16:26:28	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-fail-predictor-cb799f68	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng
16:26:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8	Unhealthy	Readiness probe failed: Get "https://10.132.0.55:8643/healthz": dial tcp 10.132.0.55:8643: connect: connection refused
16:26:28	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-fail-predictor-cb799f68 from 0 to 1
16:26:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-fail": the object has been modified; please apply your changes to the latest version and try again
16:26:29	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng	AddedInterface	Add eth0 [10.132.0.56/23] from ovn-kubernetes
16:26:30	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-kt7kv	Pulled	Successfully pulled image "kserve/paddleserver:latest" in 11.091s (11.091s including waiting). Image size: 1203659522 bytes.
16:26:30	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-kt7kv	Created	Created container: kserve-container
16:26:30	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-kt7kv	Started	Started container kube-rbac-proxy
16:26:30	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-kt7kv	Started	Started container kserve-container
16:26:30	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-kt7kv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:26:30	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-kt7kv	Created	Created container: kube-rbac-proxy
16:26:35 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:26:35 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng	Started	Started container storage-initializer
16:26:35 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng	Created	Created container: storage-initializer
16:26:35 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:26:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng	Killing	Stopping container storage-initializer
16:26:39	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd from 0 to 1
16:26:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-pass": the object has been modified; please apply your changes to the latest version and try again
16:26:39	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t
16:26:40	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t	AddedInterface	Add eth0 [10.133.0.49/23] from ovn-kubernetes
16:26:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t	Created	Created container: storage-initializer
16:26:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t	Started	Started container storage-initializer
16:26:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:26:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t	Started	Started container kserve-container
16:26:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:26:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" already present on machine
16:26:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t	Created	Created container: kube-rbac-proxy
16:26:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t	Started	Started container kube-rbac-proxy
16:26:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t	Created	Created container: kserve-container
16:26:57 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:26:57 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:27:09 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:27:09 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:27:13	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:27:13	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:27:17 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:27:17	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	InferenceServiceReady	InferenceService [isvc-paddle-runtime] is Ready
16:27:24	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-kt7kv	Killing	Stopping container kube-rbac-proxy
16:27:24	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-kt7kv	Killing	Stopping container kserve-container
16:27:27	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-kt7kv	Unhealthy	Readiness probe failed: Get "https://10.133.0.48:8643/healthz": dial tcp 10.133.0.48:8643: connect: connection refused
16:27:27 (x7)	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-kt7kv	Unhealthy	Readiness probe failed: dial tcp 10.133.0.48:8080: connect: connection refused
16:27:29	kserve-ci-e2e-test	deployment-controller	isvc-paddle-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-v2-kserve-predictor-64d4c74b4d from 0 to 1
16:27:29	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-v2-kserve-predictor-64d4c74b4d	SuccessfulCreate	Created pod: isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4
16:27:29	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
16:27:30	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:27:30	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4	Started	Started container storage-initializer
16:27:30	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4	Created	Created container: storage-initializer
16:27:30	kserve-ci-e2e-test	multus	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4	AddedInterface	Add eth0 [10.132.0.57/23] from ovn-kubernetes
16:27:35	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4	Created	Created container: kube-rbac-proxy
16:27:35	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:27:35	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4	Started	Started container kserve-container
16:27:35	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4	Created	Created container: kserve-container
16:27:35	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4	Pulled	Container image "kserve/paddleserver:latest" already present on machine
16:27:35	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4	Started	Started container kube-rbac-proxy
16:27:54 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:27:54 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:28:00 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:28:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-custom-pass] is Ready
16:28:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t	Killing	Stopping container kube-rbac-proxy
16:28:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t	Killing	Stopping container kserve-container
16:28:10	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d from 0 to 1
16:28:10 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t	Unhealthy	Readiness probe failed: dial tcp 10.133.0.49:8080: connect: connection refused
16:28:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-fail": the object has been modified; please apply your changes to the latest version and try again
16:28:10	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr
16:28:11	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr	AddedInterface	Add eth0 [10.132.0.58/23] from ovn-kubernetes
16:28:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t	Unhealthy	Readiness probe failed: Get "https://10.133.0.49:8643/healthz": dial tcp 10.133.0.49:8643: connect: connection refused
16:28:14 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:28:14 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:28:16 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:28:16 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr	Created	Created container: storage-initializer
16:28:16 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:28:16 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr	Started	Started container storage-initializer
16:28:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr	Killing	Stopping container storage-initializer
16:28:21	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-pass": the object has been modified; please apply your changes to the latest version and try again
16:28:21	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5 from 0 to 1
16:28:22	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng	AddedInterface	Add eth0 [10.133.0.50/23] from ovn-kubernetes
16:28:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	InferenceServiceReady	InferenceService [isvc-paddle-v2-kserve] is Ready
16:28:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng	Started	Started container storage-initializer
16:28:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng	Created	Created container: storage-initializer
16:28:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:28:22	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng
16:28:22 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:28:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng	Created	Created container: kserve-container
16:28:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" already present on machine
16:28:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng	Started	Started container kube-rbac-proxy
16:28:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng	Created	Created container: kube-rbac-proxy
16:28:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:28:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng	Started	Started container kserve-container
16:28:29	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:28:29	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:28:31	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4	Killing	Stopping container kserve-container
16:28:31	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4	Killing	Stopping container kube-rbac-proxy
16:28:32 (x7)	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4	Unhealthy	Readiness probe failed: dial tcp 10.132.0.57:8080: connect: connection refused
16:28:32	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4	Unhealthy	Readiness probe failed: Get "https://10.132.0.57:8643/healthz": dial tcp 10.132.0.57:8643: connect: connection refused
16:28:36	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tq9pm	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-pmml-predictor-serving-cert" not found
16:28:36	kserve-ci-e2e-test	deployment-controller	isvc-pmml-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-predictor-8674795f59 from 0 to 1
16:28:36	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-predictor-8674795f59	SuccessfulCreate	Created pod: isvc-pmml-predictor-8674795f59-tq9pm
16:28:36	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	UpdateFailed	Failed to update status for InferenceService "isvc-pmml": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml": the object has been modified; please apply your changes to the latest version and try again
16:28:37	kserve-ci-e2e-test	multus	isvc-pmml-predictor-8674795f59-tq9pm	AddedInterface	Add eth0 [10.132.0.59/23] from ovn-kubernetes
16:28:37	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tq9pm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:28:37	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tq9pm	Created	Created container: storage-initializer
16:28:37	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tq9pm	Started	Started container storage-initializer
16:28:41	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tq9pm	Pulling	Pulling image "kserve/pmmlserver:latest"
16:28:49	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tq9pm	Created	Created container: kserve-container
16:28:49	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tq9pm	Started	Started container kube-rbac-proxy
16:28:49	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tq9pm	Pulled	Successfully pulled image "kserve/pmmlserver:latest" in 7.46s (7.46s including waiting). Image size: 852995967 bytes.
16:28:49	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tq9pm	Started	Started container kserve-container
16:28:49	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tq9pm	Created	Created container: kube-rbac-proxy
16:28:49	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tq9pm	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:28:51	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:28:51	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:28:52 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:28:52 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:29:30 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng	Unhealthy	Readiness probe failed: dial tcp 10.133.0.50:8080: connect: connection refused
16:29:37 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:29:37 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:29:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-serving-pass] is Ready
16:29:41 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:29:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng	Killing	Stopping container kserve-container
16:29:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng	Killing	Stopping container kube-rbac-proxy
16:29:43	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-fail": the object has been modified; please apply your changes to the latest version and try again
16:29:43	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph	AddedInterface	Add eth0 [10.133.0.51/23] from ovn-kubernetes
16:29:43	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph
16:29:43	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9 from 0 to 1
16:29:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng	Unhealthy	Readiness probe failed: Get "https://10.133.0.50:8643/healthz": dial tcp 10.133.0.50:8643: connect: connection refused
16:29:50 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph	Created	Created container: storage-initializer
16:29:50 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph	Started	Started container storage-initializer
16:29:50 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:29:50 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:29:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph	Killing	Stopping container storage-initializer
16:29:54	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
16:29:54	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-v2-kserve-predictor-84cb55486c	SuccessfulCreate	Created pod: isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp
16:29:54	kserve-ci-e2e-test	deployment-controller	isvc-pmml-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-v2-kserve-predictor-84cb55486c from 0 to 1
16:29:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
16:29:55	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp	Created	Created container: storage-initializer
16:29:55	kserve-ci-e2e-test	multus	isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp	AddedInterface	Add eth0 [10.133.0.52/23] from ovn-kubernetes
16:29:55	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:29:55	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp	Started	Started container storage-initializer
16:30:00	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp	Pulling	Pulling image "kserve/pmmlserver:latest"
16:30:06 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tq9pm	Unhealthy	Readiness probe failed: dial tcp 10.132.0.59:8080: connect: connection refused
16:30:07	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp	Created	Created container: kube-rbac-proxy
16:30:07	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:30:07	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp	Started	Started container kserve-container
16:30:07	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp	Created	Created container: kserve-container
16:30:07	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp	Pulled	Successfully pulled image "kserve/pmmlserver:latest" in 7.312s (7.312s including waiting). Image size: 852995967 bytes.
16:30:07	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp	Started	Started container kube-rbac-proxy
16:30:16 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:30:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	InferenceServiceReady	InferenceService [isvc-pmml] is Ready
16:30:21 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:30:21 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:30:28	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tq9pm	Killing	Stopping container kube-rbac-proxy
16:30:28	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tq9pm	Killing	Stopping container kserve-container
16:30:31	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tq9pm	Unhealthy	Readiness probe failed: Get "https://10.132.0.59:8643/healthz": dial tcp 10.132.0.59:8643: connect: connection refused
16:30:33	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-txjc6	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-pmml-runtime-predictor-serving-cert" not found
16:30:33	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-runtime-predictor-885fc8777	SuccessfulCreate	Created pod: isvc-pmml-runtime-predictor-885fc8777-txjc6
16:30:33	kserve-ci-e2e-test	deployment-controller	isvc-pmml-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-runtime-predictor-885fc8777 from 0 to 1
16:30:33	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-runtime": the object has been modified; please apply your changes to the latest version and try again
16:30:34	kserve-ci-e2e-test	multus	isvc-pmml-runtime-predictor-885fc8777-txjc6	AddedInterface	Add eth0 [10.132.0.60/23] from ovn-kubernetes
16:30:34	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-txjc6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:30:34	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-txjc6	Started	Started container storage-initializer
16:30:34	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-txjc6	Created	Created container: storage-initializer
16:30:39	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-txjc6	Created	Created container: kube-rbac-proxy
16:30:39	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-txjc6	Started	Started container kube-rbac-proxy
16:30:39	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-txjc6	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:30:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:30:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:30:39	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-txjc6	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
16:30:39	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-txjc6	Created	Created container: kserve-container
16:30:39	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-txjc6	Started	Started container kserve-container
16:31:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:31:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:31:36	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	InferenceServiceReady	InferenceService [isvc-pmml-v2-kserve] is Ready
16:31:36 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:31:39 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:31:39 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:31:45	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp	Killing	Stopping container kserve-container
16:31:45	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp	Killing	Stopping container kube-rbac-proxy
16:31:46 (x12)	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp	Unhealthy	Readiness probe failed: dial tcp 10.133.0.52:8080: connect: connection refused
16:31:51	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp	Unhealthy	Readiness probe failed: Get "https://10.133.0.52:8643/healthz": context deadline exceeded
16:32:00 (x11)	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-txjc6	Unhealthy	Readiness probe failed: dial tcp 10.132.0.60:8080: connect: connection refused
16:32:03 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:32:03 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:32:10 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:32:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	InferenceServiceReady	InferenceService [isvc-pmml-runtime] is Ready
16:32:15	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-txjc6	Killing	Stopping container kube-rbac-proxy
16:32:15	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-txjc6	Killing	Stopping container kserve-container
16:32:16	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-txjc6	Unhealthy	Readiness probe failed: Get "https://10.132.0.60:8643/healthz": dial tcp 10.132.0.60:8643: connect: connection refused
16:32:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-859eae	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-859eae": the object has been modified; please apply your changes to the latest version and try again
16:32:20	kserve-ci-e2e-test	deployment-controller	isvc-primary-859eae-predictor	ScalingReplicaSet	Scaled up replica set isvc-primary-859eae-predictor-684899d844 from 0 to 1
16:32:20	kserve-ci-e2e-test	replicaset-controller	isvc-primary-859eae-predictor-684899d844	SuccessfulCreate	Created pod: isvc-primary-859eae-predictor-684899d844-qpnjm
16:32:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-859eae	UpdateFailed	Failed to update status for InferenceService "isvc-primary-859eae": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-859eae": the object has been modified; please apply your changes to the latest version and try again
16:32:21	kserve-ci-e2e-test	kubelet	isvc-primary-859eae-predictor-684899d844-qpnjm	Started	Started container storage-initializer
16:32:21	kserve-ci-e2e-test	kubelet	isvc-primary-859eae-predictor-684899d844-qpnjm	Created	Created container: storage-initializer
16:32:21	kserve-ci-e2e-test	kubelet	isvc-primary-859eae-predictor-684899d844-qpnjm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:32:21	kserve-ci-e2e-test	multus	isvc-primary-859eae-predictor-684899d844-qpnjm	AddedInterface	Add eth0 [10.133.0.53/23] from ovn-kubernetes
16:32:25	kserve-ci-e2e-test	kubelet	isvc-primary-859eae-predictor-684899d844-qpnjm	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1996" already present on machine
16:32:25	kserve-ci-e2e-test	kubelet	isvc-primary-859eae-predictor-684899d844-qpnjm	Created	Created container: kserve-container
16:32:26	kserve-ci-e2e-test	kubelet	isvc-primary-859eae-predictor-684899d844-qpnjm	Started	Started container kserve-container
16:32:26	kserve-ci-e2e-test	kubelet	isvc-primary-859eae-predictor-684899d844-qpnjm	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:32:26	kserve-ci-e2e-test	kubelet	isvc-primary-859eae-predictor-684899d844-qpnjm	Created	Created container: kube-rbac-proxy
16:32:26	kserve-ci-e2e-test	kubelet	isvc-primary-859eae-predictor-684899d844-qpnjm	Started	Started container kube-rbac-proxy
16:32:50 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-859eae-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:32:50 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-859eae-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:33:32 (x9)	kserve-ci-e2e-test	kubelet	isvc-primary-859eae-predictor-684899d844-qpnjm	Unhealthy	Readiness probe failed: dial tcp 10.133.0.53:8080: connect: connection refused
16:33:43	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-859eae	InferenceServiceReady	InferenceService [isvc-primary-859eae] is Ready
16:33:43 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-859eae	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:34:05	kserve-ci-e2e-test	replicaset-controller	isvc-secondary-859eae-predictor-54d5c46d5f	SuccessfulCreate	Created pod: isvc-secondary-859eae-predictor-54d5c46d5f-blvk6
16:34:05 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-859eae-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:34:05 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-859eae-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:34:05 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-859eae	UpdateFailed	Failed to update status for InferenceService "isvc-secondary-859eae": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-859eae": the object has been modified; please apply your changes to the latest version and try again
16:34:05	kserve-ci-e2e-test	deployment-controller	isvc-secondary-859eae-predictor	ScalingReplicaSet	Scaled up replica set isvc-secondary-859eae-predictor-54d5c46d5f from 0 to 1
16:34:05 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-859eae	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-859eae": the object has been modified; please apply your changes to the latest version and try again
16:34:06	kserve-ci-e2e-test	kubelet	isvc-secondary-859eae-predictor-54d5c46d5f-blvk6	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-secondary-859eae-predictor-serving-cert" not found
16:34:07	kserve-ci-e2e-test	multus	isvc-secondary-859eae-predictor-54d5c46d5f-blvk6	AddedInterface	Add eth0 [10.132.0.61/23] from ovn-kubernetes
16:34:13 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-859eae-predictor-54d5c46d5f-blvk6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5238b3d889a1da27166be6c0f83af4a328bdb0373db88a11a2226be3535257c4" already present on machine
16:34:13 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-859eae-predictor-54d5c46d5f-blvk6	Created	Created container: storage-initializer
16:34:13 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-859eae-predictor-54d5c46d5f-blvk6	Started	Started container storage-initializer
16:34:20	kserve-ci-e2e-test	kubelet	isvc-secondary-859eae-predictor-54d5c46d5f-blvk6	BackOff	Back-off restarting failed container storage-initializer in pod isvc-secondary-859eae-predictor-54d5c46d5f-blvk6_kserve-ci-e2e-test(cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5)
16:34:20	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-859eae-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:34:20	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-859eae-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:34:20 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-859eae	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:34:23	kserve-ci-e2e-test	kubelet	isvc-primary-859eae-predictor-684899d844-qpnjm	Killing	Stopping container kserve-container
16:34:23	kserve-ci-e2e-test	kubelet	isvc-primary-859eae-predictor-684899d844-qpnjm	Killing	Stopping container kube-rbac-proxy
16:34:27	kserve-ci-e2e-test	kubelet	isvc-primary-859eae-predictor-684899d844-qpnjm	Unhealthy	Readiness probe failed: Get "https://10.133.0.53:8643/healthz": dial tcp 10.133.0.53:8643: connect: connection refused