Events

Time	Namespace	Component	RelatedObject	Reason	Message
18:15:31	kserve-ci-e2e-test		isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h to ip-10-0-132-11.ec2.internal
18:24:41	kserve-ci-e2e-test		isvc-sklearn-runtime-predictor-599b58cf6-qw8qz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-qw8qz to ip-10-0-138-92.ec2.internal
18:38:46	kserve-ci-e2e-test		isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt to ip-10-0-132-11.ec2.internal
18:39:51	kserve-ci-e2e-test		isvc-xgboost-v2-predictor-fcc668bc-84dcd	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-84dcd to ip-10-0-138-92.ec2.internal
18:35:10	kserve-ci-e2e-test		isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5 to ip-10-0-132-11.ec2.internal
18:11:40	kserve-ci-e2e-test		isvc-predictive-sklearn-predictor-5f849d69df-ljhq9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-ljhq9 to ip-10-0-138-92.ec2.internal
17:34:59	kserve-ci-e2e-test		isvc-autogluon-v2-predictor-9684b4cd9-wt6jh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wt6jh to ip-10-0-138-92.ec2.internal
18:17:06	kserve-ci-e2e-test		isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd to ip-10-0-132-11.ec2.internal
18:37:01	kserve-ci-e2e-test		isvc-xgboost-runtime-predictor-54dd488c85-fhs64	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-fhs64 to ip-10-0-138-92.ec2.internal
18:33:25	kserve-ci-e2e-test		isvc-xgboost-predictor-7566d65bf7-wm285	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-wm285 to ip-10-0-132-11.ec2.internal
18:31:08	kserve-ci-e2e-test		isvc-triton-predictor-5d7db58676-795zp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-795zp to ip-10-0-132-11.ec2.internal
17:41:42	kserve-ci-e2e-test		isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54 to ip-10-0-138-92.ec2.internal
17:43:57	kserve-ci-e2e-test		isvc-canary-promote-predictor-d9759ffd5-m74zc	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-m74zc to ip-10-0-138-92.ec2.internal
18:30:22	kserve-ci-e2e-test		isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r to ip-10-0-138-92.ec2.internal
18:29:35	kserve-ci-e2e-test		isvc-tensorflow-predictor-547dfc5669-stxfr	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxfr to ip-10-0-132-11.ec2.internal
18:25:19	kserve-ci-e2e-test		isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6 to ip-10-0-132-11.ec2.internal
17:45:22	kserve-ci-e2e-test		isvc-canary-promote-v2-predictor-86766c4958-5mx8p	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-5mx8p to ip-10-0-132-11.ec2.internal
17:43:57	kserve-ci-e2e-test		isvc-canary-promote-v2-predictor-86766c4958-mxmmx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-mxmmx to ip-10-0-138-92.ec2.internal
18:26:24	kserve-ci-e2e-test		isvc-sklearn-v2-predictor-77f4cf5bd-z59vd	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-z59vd to ip-10-0-138-92.ec2.internal
18:28:00	kserve-ci-e2e-test		isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb to ip-10-0-132-11.ec2.internal
18:46:01	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7 to ip-10-0-138-92.ec2.internal
17:40:54	kserve-ci-e2e-test		isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5 to ip-10-0-138-92.ec2.internal
17:58:40	kserve-ci-e2e-test		isvc-mlflow-v2-runtime-predictor-6456f94494-6z977	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6z977 to ip-10-0-132-11.ec2.internal
18:23:36	kserve-ci-e2e-test		sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg	Scheduled	Successfully assigned kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg to ip-10-0-132-11.ec2.internal
17:59:35	kserve-ci-e2e-test		isvc-sklearn-mcp-predictor-5996957568-pgdxp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-pgdxp to ip-10-0-132-11.ec2.internal
18:44:19	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55 to ip-10-0-138-92.ec2.internal
18:45:50	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2 to ip-10-0-138-92.ec2.internal
18:41:16	kserve-ci-e2e-test		isvc-sklearn-s3-predictor-dd446d6b9-cl7js	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-cl7js to ip-10-0-138-92.ec2.internal
18:13:36	kserve-ci-e2e-test		isvc-predictive-xgboost-predictor-695f978978-z2vth	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-z2vth to ip-10-0-132-11.ec2.internal
18:18:42	kserve-ci-e2e-test		isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr to ip-10-0-138-92.ec2.internal
17:30:03	kserve-ci-e2e-test		message-dumper-predictor-6575bcfdfd-v67qd	Scheduled	Successfully assigned kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-v67qd to ip-10-0-138-92.ec2.internal
18:22:00	kserve-ci-e2e-test		isvc-sklearn-predictor-5f4d89cc44-qhdm8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-qhdm8 to ip-10-0-132-11.ec2.internal
18:09:31	kserve-ci-e2e-test		isvc-primary-a0e545-predictor-865b97b54f-v9n79	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-primary-a0e545-predictor-865b97b54f-v9n79 to ip-10-0-132-11.ec2.internal
18:11:06	kserve-ci-e2e-test		isvc-secondary-a0e545-predictor-74b7c6689d-wgr69	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-secondary-a0e545-predictor-74b7c6689d-wgr69 to ip-10-0-138-92.ec2.internal
17:39:03	kserve-ci-e2e-test		isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx to ip-10-0-138-92.ec2.internal
17:26:42	kserve-ci-e2e-test		isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb to ip-10-0-138-92.ec2.internal
17:47:18	kserve-ci-e2e-test		isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z to ip-10-0-132-11.ec2.internal
17:28:27	kserve-ci-e2e-test		isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8 to ip-10-0-138-92.ec2.internal
17:48:43	kserve-ci-e2e-test		isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw to ip-10-0-138-92.ec2.internal
17:47:18	kserve-ci-e2e-test		isvc-canary-rollback-v2-predictor-84f48b7656-klkls	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-klkls to ip-10-0-138-92.ec2.internal
17:50:18	kserve-ci-e2e-test		isvc-canary-stop-predictor-86dc9fdb65-vh2l8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-vh2l8 to ip-10-0-138-92.ec2.internal
17:50:18	kserve-ci-e2e-test		isvc-canary-stop-v2-predictor-77f778d864-h2x8f	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-h2x8f to ip-10-0-138-92.ec2.internal
18:11:25	kserve-ci-e2e-test		isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb to ip-10-0-132-11.ec2.internal
17:31:58	kserve-ci-e2e-test		isvc-autogluon-v1-predictor-574b5b7495-h2r48	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-h2r48 to ip-10-0-138-92.ec2.internal
18:44:08	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5 to ip-10-0-132-11.ec2.internal
17:37:01	kserve-ci-e2e-test		isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8 to ip-10-0-138-92.ec2.internal
17:51:59	kserve-ci-e2e-test		isvc-lightgbm-predictor-54f85ddf88-cx5mp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-cx5mp to ip-10-0-132-11.ec2.internal
17:53:34	kserve-ci-e2e-test		isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r to ip-10-0-138-92.ec2.internal
17:58:15	kserve-ci-e2e-test		isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt to ip-10-0-132-11.ec2.internal
17:55:10	kserve-ci-e2e-test		isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm to ip-10-0-132-11.ec2.internal
17:30:23	kserve-ci-e2e-test		isvc-logger-predictor-6b6746bb7-9l9d4	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-9l9d4 to ip-10-0-138-92.ec2.internal
17:42:22	kserve-ci-e2e-test		isvc-canary-create-predictor-cb85cf55c-7jhbv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-7jhbv to ip-10-0-138-92.ec2.internal
18:47:22	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf to ip-10-0-138-92.ec2.internal
18:01:01	kserve-ci-e2e-test		isvc-paddle-predictor-6d5c9fd4f6-shpq6	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-shpq6 to ip-10-0-138-92.ec2.internal
18:02:18	kserve-ci-e2e-test		isvc-paddle-runtime-predictor-58b99fff76-cvnvf	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cvnvf to ip-10-0-138-92.ec2.internal
18:03:24	kserve-ci-e2e-test		isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5 to ip-10-0-138-92.ec2.internal
18:04:32	kserve-ci-e2e-test		isvc-pmml-predictor-8674795f59-tpxjh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tpxjh to ip-10-0-132-11.ec2.internal
18:06:18	kserve-ci-e2e-test		isvc-pmml-runtime-predictor-885fc8777-87tld	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-87tld to ip-10-0-132-11.ec2.internal
18:07:54	kserve-ci-e2e-test		isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf to ip-10-0-132-11.ec2.internal
17:42:22	kserve-ci-e2e-test		isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc to ip-10-0-138-92.ec2.internal
18:20:18	kserve-ci-e2e-test		isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl to ip-10-0-132-11.ec2.internal
18:42:37	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658 to ip-10-0-138-92.ec2.internal
18:36:05	kserve-ci-e2e-test		xgboost-v2-mlserver-predictor-59944bcf4c-xq85r	Scheduled	Successfully assigned kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xq85r to ip-10-0-132-11.ec2.internal
17:26:42	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-predictor-86cb8f94cb from 0 to 1
17:26:42 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
17:26:42 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
17:26:42	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-predictor-86cb8f94cb	SuccessfulCreate	Created pod: isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb
17:26:43	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb	AddedInterface	Add eth0 [10.133.0.28/23] from ovn-kubernetes
17:26:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
17:26:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb	Created	Created container: storage-initializer
17:26:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb	Started	Started container storage-initializer
17:26:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814"
17:27:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" in 13.593s (13.593s including waiting). Image size: 1555877302 bytes.
17:27:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb	Started	Started container kserve-container
17:27:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
17:27:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb	Created	Created container: kserve-container
17:27:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.214s (2.214s including waiting). Image size: 211946088 bytes.
17:27:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb	Created	Created container: kube-rbac-proxy
17:27:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb	Started	Started container kube-rbac-proxy
17:27:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb	Pulling	Pulling image "quay.io/opendatahub/kserve-agent@sha256:efaf0bd4be5e59e0fd5df072416f565ae616956e4a23f814ea31d63b5efa6d4b"
17:27:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:efaf0bd4be5e59e0fd5df072416f565ae616956e4a23f814ea31d63b5efa6d4b" in 2.673s (2.673s including waiting). Image size: 240168571 bytes.
17:27:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb	Started	Started container agent
17:27:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb	Created	Created container: agent
17:27:42 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:27:42 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:28:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InferenceServiceReady	InferenceService [isvc-sklearn-batcher] is Ready
17:28:19 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:28:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb	Killing	Stopping container kube-rbac-proxy
17:28:27	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-custom-predictor-55c4fb9c75	SuccessfulCreate	Created pod: isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8
17:28:27 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:28:27 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:28:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb	Killing	Stopping container agent
17:28:27	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher-custom": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher-custom": the object has been modified; please apply your changes to the latest version and try again
17:28:27	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-custom-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-custom-predictor-55c4fb9c75 from 0 to 1
17:28:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb	Killing	Stopping container kserve-container
17:28:28	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8	AddedInterface	Add eth0 [10.133.0.29/23] from ovn-kubernetes
17:28:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8	Created	Created container: storage-initializer
17:28:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
17:28:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8	Started	Started container storage-initializer
17:28:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine
17:28:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8	Created	Created container: kserve-container
17:28:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8	Started	Started container kserve-container
17:28:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:28:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:efaf0bd4be5e59e0fd5df072416f565ae616956e4a23f814ea31d63b5efa6d4b" already present on machine
17:28:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8	Created	Created container: agent
17:28:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8	Started	Started container kube-rbac-proxy
17:28:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8	Created	Created container: kube-rbac-proxy
17:28:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8	Started	Started container agent
17:28:38 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb	Unhealthy	Readiness probe failed: Get "https://10.133.0.28:8643/healthz": dial tcp 10.133.0.28:8643: connect: connection refused
17:28:39 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
17:28:39 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-86cb8f94cb-cnhdb	Unhealthy	Readiness probe failed: dial tcp 10.133.0.28:8080: connect: connection refused
17:29:12 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:29:12 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:29:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	InferenceServiceReady	InferenceService [isvc-sklearn-batcher-custom] is Ready
17:29:50 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:29:57 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:29:57 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:30:03	kserve-ci-e2e-test	replicaset-controller	message-dumper-predictor-6575bcfdfd	SuccessfulCreate	Created pod: message-dumper-predictor-6575bcfdfd-v67qd
17:30:03 (x2)	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	UpdateFailed	Failed to update status for InferenceService "message-dumper": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper": the object has been modified; please apply your changes to the latest version and try again
17:30:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8	Killing	Stopping container kserve-container
17:30:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8	Killing	Stopping container kube-rbac-proxy
17:30:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8	Killing	Stopping container agent
17:30:03	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-v67qd	Pulling	Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display"
17:30:03	kserve-ci-e2e-test	deployment-controller	message-dumper-predictor	ScalingReplicaSet	Scaled up replica set message-dumper-predictor-6575bcfdfd from 0 to 1
17:30:03	kserve-ci-e2e-test	multus	message-dumper-predictor-6575bcfdfd-v67qd	AddedInterface	Add eth0 [10.133.0.30/23] from ovn-kubernetes
17:30:03	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper": the object has been modified; please apply your changes to the latest version and try again
17:30:04	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-v67qd	Pulled	Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.248s (1.248s including waiting). Image size: 14813193 bytes.
17:30:04	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-v67qd	Created	Created container: kserve-container
17:30:04	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-v67qd	Started	Started container kserve-container
17:30:04	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-v67qd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:30:04	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-v67qd	Created	Created container: kube-rbac-proxy
17:30:04	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-v67qd	Started	Started container kube-rbac-proxy
17:30:09 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8	Unhealthy	Readiness probe failed: dial tcp 10.133.0.29:5000: connect: connection refused
17:30:09 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
17:30:13 (x8)	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:30:13	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	InferenceServiceReady	InferenceService [message-dumper] is Ready
17:30:19 (x4)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-55c4fb9c75-jlfz8	Unhealthy	Readiness probe failed: Get "https://10.133.0.29:8643/healthz": dial tcp 10.133.0.29:8643: connect: connection refused
17:30:23	kserve-ci-e2e-test	replicaset-controller	isvc-logger-predictor-6b6746bb7	SuccessfulCreate	Created pod: isvc-logger-predictor-6b6746bb7-9l9d4
17:30:23	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b6746bb7-9l9d4	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-logger-predictor-serving-cert" not found
17:30:23	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	UpdateFailed	Failed to update status for InferenceService "isvc-logger": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger": the object has been modified; please apply your changes to the latest version and try again
17:30:23	kserve-ci-e2e-test	deployment-controller	isvc-logger-predictor	ScalingReplicaSet	Scaled up replica set isvc-logger-predictor-6b6746bb7 from 0 to 1
17:30:24	kserve-ci-e2e-test	multus	isvc-logger-predictor-6b6746bb7-9l9d4	AddedInterface	Add eth0 [10.133.0.31/23] from ovn-kubernetes
17:30:24	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b6746bb7-9l9d4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
17:30:24	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b6746bb7-9l9d4	Created	Created container: storage-initializer
17:30:24	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b6746bb7-9l9d4	Started	Started container storage-initializer
17:30:28	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b6746bb7-9l9d4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:30:28	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b6746bb7-9l9d4	Started	Started container kserve-container
17:30:28	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b6746bb7-9l9d4	Created	Created container: kube-rbac-proxy
17:30:28	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b6746bb7-9l9d4	Started	Started container kube-rbac-proxy
17:30:28	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b6746bb7-9l9d4	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:efaf0bd4be5e59e0fd5df072416f565ae616956e4a23f814ea31d63b5efa6d4b" already present on machine
17:30:28	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b6746bb7-9l9d4	Created	Created container: agent
17:30:28	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b6746bb7-9l9d4	Started	Started container agent
17:30:28	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b6746bb7-9l9d4	Created	Created container: kserve-container
17:30:28	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b6746bb7-9l9d4	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine
17:30:33 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:30:33 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:30:53 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:30:53 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:31:45	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	InferenceServiceReady	InferenceService [isvc-logger] is Ready
17:31:45 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:31:53 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:31:53 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:31:58	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v1": the object has been modified; please apply your changes to the latest version and try again
17:31:58	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-h2r48	Started	Started container storage-initializer
17:31:58	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-h2r48	Created	Created container: storage-initializer
17:31:58	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-h2r48	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
17:31:58	kserve-ci-e2e-test	multus	isvc-autogluon-v1-predictor-574b5b7495-h2r48	AddedInterface	Add eth0 [10.133.0.32/23] from ovn-kubernetes
17:31:58	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b6746bb7-9l9d4	Killing	Stopping container kube-rbac-proxy
17:31:58	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-v67qd	Killing	Stopping container kube-rbac-proxy
17:31:58	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-v67qd	Killing	Stopping container kserve-container
17:31:58	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v1-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v1-predictor-574b5b7495 from 0 to 1
17:31:58	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v1-predictor-574b5b7495	SuccessfulCreate	Created pod: isvc-autogluon-v1-predictor-574b5b7495-h2r48
17:31:58	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b6746bb7-9l9d4	Killing	Stopping container kserve-container
17:31:58	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b6746bb7-9l9d4	Killing	Stopping container agent
17:32:05 (x10)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b6746bb7-9l9d4	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
17:32:05 (x10)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b6746bb7-9l9d4	Unhealthy	Readiness probe failed: dial tcp 10.133.0.31:8080: connect: connection refused
17:32:15 (x4)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6b6746bb7-9l9d4	Unhealthy	Readiness probe failed: Get "https://10.133.0.31:8643/healthz": dial tcp 10.133.0.31:8643: connect: connection refused
17:32:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-h2r48	Pulling	Pulling image "kserve/autogluonserver:latest"
17:34:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-h2r48	Created	Created container: kserve-container
17:34:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-h2r48	Pulled	Successfully pulled image "kserve/autogluonserver:latest" in 1m59.84s (1m59.84s including waiting). Image size: 8940333845 bytes.
17:34:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-h2r48	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:34:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-h2r48	Started	Started container kserve-container
17:34:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-h2r48	Created	Created container: kube-rbac-proxy
17:34:29	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-h2r48	Started	Started container kube-rbac-proxy
17:34:43 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:34:43 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:34:45	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	InferenceServiceReady	InferenceService [isvc-autogluon-v1] is Ready
17:34:45 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:34:54	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-h2r48	Killing	Stopping container kube-rbac-proxy
17:34:54	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-h2r48	Killing	Stopping container kserve-container
17:34:55 (x4)	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-h2r48	Unhealthy	Readiness probe failed: dial tcp 10.133.0.32:8080: connect: connection refused
17:34:55	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-h2r48	Unhealthy	Readiness probe failed: Get "https://10.133.0.32:8643/healthz": dial tcp 10.133.0.32:8643: connect: connection refused
17:34:59	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-predictor-9684b4cd9 from 0 to 1
17:34:59	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2": the object has been modified; please apply your changes to the latest version and try again
17:34:59	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-predictor-9684b4cd9	SuccessfulCreate	Created pod: isvc-autogluon-v2-predictor-9684b4cd9-wt6jh
17:35:00	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-wt6jh	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-autogluon-v2-predictor-serving-cert" not found
17:35:01	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-wt6jh	Started	Started container storage-initializer
17:35:01	kserve-ci-e2e-test	multus	isvc-autogluon-v2-predictor-9684b4cd9-wt6jh	AddedInterface	Add eth0 [10.133.0.33/23] from ovn-kubernetes
17:35:01	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-wt6jh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
17:35:01	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-wt6jh	Created	Created container: storage-initializer
17:35:06	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-wt6jh	Started	Started container kserve-container
17:35:06	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-wt6jh	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
17:35:06	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-wt6jh	Created	Created container: kserve-container
17:35:06	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-wt6jh	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:35:06	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-wt6jh	Created	Created container: kube-rbac-proxy
17:35:06	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-wt6jh	Started	Started container kube-rbac-proxy
17:35:44 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:35:44 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:36:43 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:36:43	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	InferenceServiceReady	InferenceService [isvc-autogluon-v2] is Ready
17:36:44 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:36:44 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:36:54	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-wt6jh	Unhealthy	Readiness probe failed: Get "http://10.133.0.33:8080/v2/models/isvc-autogluon-v2/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
17:36:56	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-wt6jh	Killing	Stopping container kserve-container
17:36:56	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-wt6jh	Killing	Stopping container kube-rbac-proxy
17:37:01	kserve-ci-e2e-test	multus	isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8	AddedInterface	Add eth0 [10.133.0.34/23] from ovn-kubernetes
17:37:01	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-variants-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-variants-predictor-5c889d8844 from 0 to 1
17:37:01	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-variants-predictor-5c889d8844	SuccessfulCreate	Created pod: isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8
17:37:01	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
17:37:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2-variants": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-variants": the object has been modified; please apply your changes to the latest version and try again
17:37:01	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8	Started	Started container storage-initializer
17:37:01	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8	Created	Created container: storage-initializer
17:37:07	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8	Started	Started container kube-rbac-proxy
17:37:07	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8	Created	Created container: kserve-container
17:37:07	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8	Started	Started container kserve-container
17:37:07	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:37:07	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8	Created	Created container: kube-rbac-proxy
17:37:07	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
17:37:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:37:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:38:42 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:38:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	InferenceServiceReady	InferenceService [isvc-autogluon-v2-variants] is Ready
17:38:46 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:38:46 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:38:53	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8	Unhealthy	Readiness probe failed: Get "http://10.133.0.34:8080/v2/models/isvc-autogluon-v2-variants/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
17:38:57	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8	Killing	Stopping container kube-rbac-proxy
17:38:57	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-6z2z8	Killing	Stopping container kserve-container
17:39:02	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-noslash-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-noslash-predictor-5499c79f9f from 0 to 1
17:39:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
17:39:03	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-noslash-predictor-5499c79f9f	SuccessfulCreate	Created pod: isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx
17:39:03 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2-noslash": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-noslash": the object has been modified; please apply your changes to the latest version and try again
17:39:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-noslash": the object has been modified; please apply your changes to the latest version and try again
17:39:04	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx	Started	Started container storage-initializer
17:39:04	kserve-ci-e2e-test	multus	isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx	AddedInterface	Add eth0 [10.133.0.35/23] from ovn-kubernetes
17:39:04	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
17:39:04	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx	Created	Created container: storage-initializer
17:39:09	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
17:39:09	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx	Started	Started container kserve-container
17:39:09	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:39:09	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx	Created	Created container: kube-rbac-proxy
17:39:09	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx	Created	Created container: kserve-container
17:39:09	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx	Started	Started container kube-rbac-proxy
17:39:17	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:39:17	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:40:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	InferenceServiceReady	InferenceService [isvc-autogluon-v2-noslash] is Ready
17:40:39 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:40:48 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:40:48 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:40:49	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx	Killing	Stopping container kube-rbac-proxy
17:40:49	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-6mrxx	Killing	Stopping container kserve-container
17:40:54	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-ts-v1-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-ts-v1-predictor-6cf79d9f7d from 0 to 1
17:40:54	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
17:40:54	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-ts-v1-predictor-6cf79d9f7d	SuccessfulCreate	Created pod: isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5
17:40:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-ts-v1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1": the object has been modified; please apply your changes to the latest version and try again
17:40:55	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5	Started	Started container storage-initializer
17:40:55	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
17:40:55	kserve-ci-e2e-test	multus	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5	AddedInterface	Add eth0 [10.133.0.36/23] from ovn-kubernetes
17:40:55	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5	Created	Created container: storage-initializer
17:41:04	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
17:41:04	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5	Created	Created container: kserve-container
17:41:04	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5	Created	Created container: kube-rbac-proxy
17:41:04	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:41:04	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5	Started	Started container kube-rbac-proxy
17:41:04	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5	Started	Started container kserve-container
17:41:11 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5	Unhealthy	Readiness probe failed: dial tcp 10.133.0.36:8080: connect: connection refused
17:41:21 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:41:21	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	InferenceServiceReady	InferenceService [isvc-autogluon-ts-v1] is Ready
17:41:24 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:41:24 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:41:36	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5	Killing	Stopping container kserve-container
17:41:36	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9wkc5	Killing	Stopping container kube-rbac-proxy
17:41:42	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54	Started	Started container storage-initializer
17:41:42	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54	Created	Created container: storage-initializer
17:41:42	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-ts-v1-noslash-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-ts-v1-noslash-predictor-b54fbf585 from 0 to 1
17:41:42	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585	SuccessfulCreate	Created pod: isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54
17:41:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-ts-v1-noslash": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1-noslash": the object has been modified; please apply your changes to the latest version and try again
17:41:42	kserve-ci-e2e-test	multus	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54	AddedInterface	Add eth0 [10.133.0.37/23] from ovn-kubernetes
17:41:42	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
17:41:52	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54	Started	Started container kube-rbac-proxy
17:41:52	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:41:52	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
17:41:52	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54	Created	Created container: kube-rbac-proxy
17:41:52	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54	Created	Created container: kserve-container
17:41:52	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54	Started	Started container kserve-container
17:41:59 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54	Unhealthy	Readiness probe failed: dial tcp 10.133.0.37:8080: connect: connection refused
17:42:09 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:42:09	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	InferenceServiceReady	InferenceService [isvc-autogluon-ts-v1-noslash] is Ready
17:42:12 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:42:12 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:42:17	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54	Killing	Stopping container kserve-container
17:42:17	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-94m54	Killing	Stopping container kube-rbac-proxy
17:42:22	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-create-v2-predictor-6d8d4768d4 from 0 to 1
17:42:22	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-cb85cf55c	SuccessfulCreate	Created pod: isvc-canary-create-predictor-cb85cf55c-7jhbv
17:42:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	UpdateFailed	Failed to update status for InferenceService "isvc-canary-create": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again
17:42:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again
17:42:22	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-create-predictor-cb85cf55c from 0 to 1
17:42:22	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-create-v2-predictor-serving-cert" not found
17:42:22	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-v2-predictor-6d8d4768d4	SuccessfulCreate	Created pod: isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc
17:42:22	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-cb85cf55c-7jhbv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
17:42:22	kserve-ci-e2e-test	multus	isvc-canary-create-predictor-cb85cf55c-7jhbv	AddedInterface	Add eth0 [10.133.0.38/23] from ovn-kubernetes
17:42:23	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
17:42:23	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc	Created	Created container: storage-initializer
17:42:23	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc	Started	Started container storage-initializer
17:42:23	kserve-ci-e2e-test	multus	isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc	AddedInterface	Add eth0 [10.133.0.39/23] from ovn-kubernetes
17:42:23	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-cb85cf55c-7jhbv	Started	Started container storage-initializer
17:42:23	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-cb85cf55c-7jhbv	Created	Created container: storage-initializer
17:42:27	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:42:27	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-cb85cf55c-7jhbv	Created	Created container: kserve-container
17:42:27	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc	Started	Started container kube-rbac-proxy
17:42:27	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-cb85cf55c-7jhbv	Started	Started container kube-rbac-proxy
17:42:27	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-cb85cf55c-7jhbv	Started	Started container kserve-container
17:42:27	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-cb85cf55c-7jhbv	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine
17:42:27	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc	Created	Created container: kube-rbac-proxy
17:42:27	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-cb85cf55c-7jhbv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:42:27	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc	Started	Started container kserve-container
17:42:27	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc	Created	Created container: kserve-container
17:42:27	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine
17:42:27	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-cb85cf55c-7jhbv	Created	Created container: kube-rbac-proxy
17:43:39 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc	Unhealthy	Readiness probe failed: dial tcp 10.133.0.39:8080: connect: connection refused
17:43:45	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	InferenceServiceReady	InferenceService [isvc-canary-create] is Ready
17:43:49 (x16)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:43:52	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-cb85cf55c-7jhbv	Killing	Stopping container kserve-container
17:43:52	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-cb85cf55c-7jhbv	Killing	Stopping container kube-rbac-proxy
17:43:52	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc	Killing	Stopping container kube-rbac-proxy
17:43:52	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc	Killing	Stopping container kserve-container
17:43:54	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-cb85cf55c-7jhbv	Unhealthy	Readiness probe failed: Get "https://10.133.0.38:8643/healthz": dial tcp 10.133.0.38:8643: connect: connection refused
17:43:54	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6d8d4768d4-wwxsc	Unhealthy	Readiness probe failed: Get "https://10.133.0.39:8643/healthz": dial tcp 10.133.0.39:8643: connect: connection refused
17:43:55 (x11)	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-cb85cf55c-7jhbv	Unhealthy	Readiness probe failed: dial tcp 10.133.0.38:8080: connect: connection refused
17:43:57	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-d9759ffd5	SuccessfulCreate	Created pod: isvc-canary-promote-predictor-d9759ffd5-m74zc
17:43:57	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-v2-predictor-86766c4958 from 0 to 1
17:43:57	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-v2-predictor-86766c4958	SuccessfulCreate	Created pod: isvc-canary-promote-v2-predictor-86766c4958-mxmmx
17:43:57	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-predictor-d9759ffd5 from 0 to 1
17:43:58	kserve-ci-e2e-test	multus	isvc-canary-promote-v2-predictor-86766c4958-mxmmx	AddedInterface	Add eth0 [10.133.0.41/23] from ovn-kubernetes
17:43:58	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-d9759ffd5-m74zc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
17:43:58	kserve-ci-e2e-test	multus	isvc-canary-promote-predictor-d9759ffd5-m74zc	AddedInterface	Add eth0 [10.133.0.40/23] from ovn-kubernetes
17:43:58	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-d9759ffd5-m74zc	Started	Started container storage-initializer
17:43:58	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-d9759ffd5-m74zc	Created	Created container: storage-initializer
17:43:58	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-mxmmx	Started	Started container storage-initializer
17:43:58	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-mxmmx	Created	Created container: storage-initializer
17:43:58	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-mxmmx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
17:43:58	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-mxmmx	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
17:44:02	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-d9759ffd5-m74zc	Started	Started container kserve-container
17:44:02	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-d9759ffd5-m74zc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:44:02	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-mxmmx	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine
17:44:02	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-mxmmx	Created	Created container: kserve-container
17:44:02	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-mxmmx	Started	Started container kserve-container
17:44:02	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-mxmmx	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:44:02	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-d9759ffd5-m74zc	Created	Created container: kserve-container
17:44:02	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-d9759ffd5-m74zc	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine
17:44:03	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-d9759ffd5-m74zc	Started	Started container kube-rbac-proxy
17:44:03	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-d9759ffd5-m74zc	Created	Created container: kube-rbac-proxy
17:44:03	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-mxmmx	Created	Created container: kube-rbac-proxy
17:44:03	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-mxmmx	Started	Started container kube-rbac-proxy
17:45:10 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-mxmmx	Unhealthy	Readiness probe failed: dial tcp 10.133.0.41:8080: connect: connection refused
17:45:22 (x19)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:45:22 (x3)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again
17:45:22 (x3)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	UpdateFailed	Failed to update status for InferenceService "isvc-canary-promote": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again
17:45:22	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-d9759ffd5-m74zc	Killing	Stopping container kserve-container
17:45:22	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-d9759ffd5-m74zc	Killing	Stopping container kube-rbac-proxy
17:45:22	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-v2-predictor-86766c4958 from 1 to 2
17:45:22	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-v2-predictor-86766c4958	SuccessfulCreate	Created pod: isvc-canary-promote-v2-predictor-86766c4958-5mx8p
17:45:23	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-5mx8p	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da"
17:45:23	kserve-ci-e2e-test	multus	isvc-canary-promote-v2-predictor-86766c4958-5mx8p	AddedInterface	Add eth0 [10.134.0.24/23] from ovn-kubernetes
17:45:24	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-d9759ffd5-m74zc	Unhealthy	Readiness probe failed: Get "https://10.133.0.40:8643/healthz": dial tcp 10.133.0.40:8643: connect: connection refused
17:45:24 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-d9759ffd5-m74zc	Unhealthy	Readiness probe failed: dial tcp 10.133.0.40:8080: connect: connection refused
17:45:26	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-5mx8p	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" in 3.539s (3.539s including waiting). Image size: 300915674 bytes.
17:45:26	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-5mx8p	Created	Created container: storage-initializer
17:45:26	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-5mx8p	Started	Started container storage-initializer
17:45:30	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-5mx8p	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814"
17:45:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-5mx8p	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
17:45:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-5mx8p	Started	Started container kserve-container
17:45:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-5mx8p	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" in 13.827s (13.827s including waiting). Image size: 1555877302 bytes.
17:45:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-5mx8p	Created	Created container: kserve-container
17:45:46	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-5mx8p	Started	Started container kube-rbac-proxy
17:45:46	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-5mx8p	Created	Created container: kube-rbac-proxy
17:45:46	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-5mx8p	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.231s (2.231s including waiting). Image size: 211946088 bytes.
17:47:03 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	InferenceServiceReady	InferenceService [isvc-canary-promote] is Ready
17:47:13	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-5mx8p	Unhealthy	Readiness probe failed: Get "https://10.134.0.24:8643/healthz": dial tcp 10.134.0.24:8643: connect: connection refused
17:47:13 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-5mx8p	Unhealthy	Readiness probe failed: dial tcp 10.134.0.24:8080: connect: connection refused
17:47:13	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-5mx8p	Killing	Stopping container kserve-container
17:47:13	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-5mx8p	Killing	Stopping container kube-rbac-proxy
17:47:13	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-mxmmx	Killing	Stopping container kube-rbac-proxy
17:47:13	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-mxmmx	Killing	Stopping container kserve-container
17:47:15	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-86766c4958-mxmmx	Unhealthy	Readiness probe failed: Get "https://10.133.0.41:8643/healthz": dial tcp 10.133.0.41:8643: connect: connection refused
17:47:18	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z	Created	Created container: storage-initializer
17:47:18 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	UpdateFailed	Failed to update status for InferenceService "isvc-canary-rollback": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again
17:47:18	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-84f48b7656-klkls	Started	Started container storage-initializer
17:47:18	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-84f48b7656-klkls	Created	Created container: storage-initializer
17:47:18	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-84f48b7656-klkls	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
17:47:18	kserve-ci-e2e-test	multus	isvc-canary-rollback-v2-predictor-84f48b7656-klkls	AddedInterface	Add eth0 [10.133.0.42/23] from ovn-kubernetes
17:47:18	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
17:47:18	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-v2-predictor-84f48b7656 from 0 to 1
17:47:18	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z	Started	Started container storage-initializer
17:47:18	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-5bbb8c8cfc	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z
17:47:18	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z	AddedInterface	Add eth0 [10.134.0.25/23] from ovn-kubernetes
17:47:18 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again
17:47:18	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-predictor-5bbb8c8cfc from 0 to 1
17:47:18	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-v2-predictor-84f48b7656	SuccessfulCreate	Created pod: isvc-canary-rollback-v2-predictor-84f48b7656-klkls
17:47:22	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-84f48b7656-klkls	Started	Started container kserve-container
17:47:22	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-84f48b7656-klkls	Started	Started container kube-rbac-proxy
17:47:22	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-84f48b7656-klkls	Created	Created container: kube-rbac-proxy
17:47:22	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-84f48b7656-klkls	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:47:22	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-84f48b7656-klkls	Created	Created container: kserve-container
17:47:22	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-84f48b7656-klkls	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine
17:47:23	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z	Started	Started container kube-rbac-proxy
17:47:23	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine
17:47:23	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z	Created	Created container: kserve-container
17:47:23	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z	Started	Started container kserve-container
17:47:23	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:47:23	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z	Created	Created container: kube-rbac-proxy
17:48:29 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-84f48b7656-klkls	Unhealthy	Readiness probe failed: dial tcp 10.133.0.42:8080: connect: connection refused
17:48:43	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-5bbb8c8cfc	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw
17:48:43	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-84f48b7656-klkls	Killing	Stopping container kube-rbac-proxy
17:48:43 (x20)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:48:43	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-84f48b7656-klkls	Killing	Stopping container kserve-container
17:48:43	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw	AddedInterface	Add eth0 [10.133.0.43/23] from ovn-kubernetes
17:48:43	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
17:48:43	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw	Created	Created container: storage-initializer
17:48:43	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw	Started	Started container storage-initializer
17:48:43	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-predictor-5bbb8c8cfc from 1 to 2
17:48:43	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	InferenceServiceNotReady	InferenceService [isvc-canary-rollback] is no longer Ready because of: PredictorReady, Ready, Stopped
17:48:44	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-84f48b7656-klkls	Unhealthy	Readiness probe failed: Get "https://10.133.0.42:8643/healthz": dial tcp 10.133.0.42:8643: connect: connection refused
17:48:48	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw	Started	Started container kube-rbac-proxy
17:48:48	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine
17:48:48	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw	Created	Created container: kube-rbac-proxy
17:48:48	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw	Created	Created container: kserve-container
17:48:48	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw	Started	Started container kserve-container
17:48:48	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:50:06 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	InferenceServiceReady	InferenceService [isvc-canary-rollback] is Ready
17:50:13	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw	Killing	Stopping container kserve-container
17:50:13	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw	Killing	Stopping container kube-rbac-proxy
17:50:13	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z	Killing	Stopping container kserve-container
17:50:13	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z	Killing	Stopping container kube-rbac-proxy
17:50:16	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z	Unhealthy	Readiness probe failed: Get "https://10.134.0.25:8643/healthz": dial tcp 10.134.0.25:8643: connect: connection refused
17:50:16 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw	Unhealthy	Readiness probe failed: dial tcp 10.133.0.43:8080: connect: connection refused
17:50:16	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-mvtqw	Unhealthy	Readiness probe failed: Get "https://10.133.0.43:8643/healthz": dial tcp 10.133.0.43:8643: connect: connection refused
17:50:17 (x11)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-5bbb8c8cfc-f8r5z	Unhealthy	Readiness probe failed: dial tcp 10.134.0.25:8080: connect: connection refused
17:50:18 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again
17:50:18	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-stop-predictor-86dc9fdb65 from 0 to 1
17:50:18 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	UpdateFailed	Failed to update status for InferenceService "isvc-canary-stop": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again
17:50:18	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-stop-v2-predictor-77f778d864 from 0 to 1
17:50:18	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-v2-predictor-77f778d864	SuccessfulCreate	Created pod: isvc-canary-stop-v2-predictor-77f778d864-h2x8f
17:50:18	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-86dc9fdb65	SuccessfulCreate	Created pod: isvc-canary-stop-predictor-86dc9fdb65-vh2l8
17:50:19	kserve-ci-e2e-test	multus	isvc-canary-stop-v2-predictor-77f778d864-h2x8f	AddedInterface	Add eth0 [10.133.0.45/23] from ovn-kubernetes
17:50:19	kserve-ci-e2e-test	multus	isvc-canary-stop-predictor-86dc9fdb65-vh2l8	AddedInterface	Add eth0 [10.133.0.44/23] from ovn-kubernetes
17:50:19	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-77f778d864-h2x8f	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
17:50:19	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-77f778d864-h2x8f	Created	Created container: storage-initializer
17:50:19	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-77f778d864-h2x8f	Started	Started container storage-initializer
17:50:19	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-86dc9fdb65-vh2l8	Started	Started container storage-initializer
17:50:19	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-86dc9fdb65-vh2l8	Created	Created container: storage-initializer
17:50:19	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-86dc9fdb65-vh2l8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
17:50:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-77f778d864-h2x8f	Started	Started container kserve-container
17:50:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-86dc9fdb65-vh2l8	Created	Created container: kserve-container
17:50:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-86dc9fdb65-vh2l8	Started	Started container kserve-container
17:50:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-86dc9fdb65-vh2l8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:50:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-86dc9fdb65-vh2l8	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine
17:50:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-86dc9fdb65-vh2l8	Created	Created container: kube-rbac-proxy
17:50:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-86dc9fdb65-vh2l8	Started	Started container kube-rbac-proxy
17:50:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-77f778d864-h2x8f	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine
17:50:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-77f778d864-h2x8f	Created	Created container: kserve-container
17:50:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-77f778d864-h2x8f	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:50:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-77f778d864-h2x8f	Created	Created container: kube-rbac-proxy
17:50:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-77f778d864-h2x8f	Started	Started container kube-rbac-proxy
17:51:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	InferenceServiceReady	InferenceService [isvc-canary-stop] is Ready
17:51:48	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-77f778d864-h2x8f	Killing	Stopping container kserve-container
17:51:48 (x18)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:51:48	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	InferenceServiceNotReady	InferenceService [isvc-canary-stop] is no longer Ready because of: PredictorReady, Ready
17:51:48	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-77f778d864-h2x8f	Killing	Stopping container kube-rbac-proxy
17:51:48	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-86dc9fdb65-vh2l8	Killing	Stopping container kube-rbac-proxy
17:51:48	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-86dc9fdb65-vh2l8	Killing	Stopping container kserve-container
17:51:50 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-86dc9fdb65-vh2l8	Unhealthy	Readiness probe failed: dial tcp 10.133.0.44:8080: connect: connection refused
17:51:50 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-77f778d864-h2x8f	Unhealthy	Readiness probe failed: dial tcp 10.133.0.45:8080: connect: connection refused
17:51:50	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-86dc9fdb65-vh2l8	Unhealthy	Readiness probe failed: Get "https://10.133.0.44:8643/healthz": dial tcp 10.133.0.44:8643: connect: connection refused
17:51:50	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-77f778d864-h2x8f	Unhealthy	Readiness probe failed: Get "https://10.133.0.45:8643/healthz": dial tcp 10.133.0.45:8643: connect: connection refused
17:51:59 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm": the object has been modified; please apply your changes to the latest version and try again
17:51:59	kserve-ci-e2e-test	multus	isvc-lightgbm-predictor-54f85ddf88-cx5mp	AddedInterface	Add eth0 [10.134.0.26/23] from ovn-kubernetes
17:51:59	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-cx5mp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
17:51:59	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-cx5mp	Started	Started container storage-initializer
17:51:59	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm": the object has been modified; please apply your changes to the latest version and try again
17:51:59	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-cx5mp	Created	Created container: storage-initializer
17:51:59	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-predictor-54f85ddf88	SuccessfulCreate	Created pod: isvc-lightgbm-predictor-54f85ddf88-cx5mp
17:51:59	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-predictor-54f85ddf88 from 0 to 1
17:52:03	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-cx5mp	Pulling	Pulling image "kserve/lgbserver:latest"
17:52:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-cx5mp	Started	Started container kube-rbac-proxy
17:52:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-cx5mp	Started	Started container kserve-container
17:52:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-cx5mp	Created	Created container: kserve-container
17:52:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-cx5mp	Pulled	Successfully pulled image "kserve/lgbserver:latest" in 6.53s (6.53s including waiting). Image size: 607170829 bytes.
17:52:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-cx5mp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:52:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-cx5mp	Created	Created container: kube-rbac-proxy
17:52:44 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:52:44 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:53:14 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:53:14 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:53:16 (x9)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-cx5mp	Unhealthy	Readiness probe failed: dial tcp 10.134.0.26:8080: connect: connection refused
17:53:26 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:53:26	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	InferenceServiceReady	InferenceService [isvc-lightgbm] is Ready
17:53:29	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-cx5mp	Killing	Stopping container kserve-container
17:53:29	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-cx5mp	Killing	Stopping container kube-rbac-proxy
17:53:31	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-cx5mp	Unhealthy	Readiness probe failed: Get "https://10.134.0.26:8643/healthz": dial tcp 10.134.0.26:8643: connect: connection refused
17:53:34	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-runtime-predictor-577f7db7cb from 0 to 1
17:53:34	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-runtime-predictor-577f7db7cb	SuccessfulCreate	Created pod: isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r
17:53:34 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-runtime": the object has been modified; please apply your changes to the latest version and try again
17:53:34	kserve-ci-e2e-test	multus	isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r	AddedInterface	Add eth0 [10.133.0.46/23] from ovn-kubernetes
17:53:34	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
17:53:34	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r	Created	Created container: storage-initializer
17:53:34	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r	Started	Started container storage-initializer
17:53:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-runtime": the object has been modified; please apply your changes to the latest version and try again
17:53:39	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r	Pulling	Pulling image "kserve/lgbserver:latest"
17:53:45	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r	Started	Started container kserve-container
17:53:45	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r	Created	Created container: kserve-container
17:53:45	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r	Pulled	Successfully pulled image "kserve/lgbserver:latest" in 6.293s (6.293s including waiting). Image size: 607170829 bytes.
17:53:45	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:53:45	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r	Created	Created container: kube-rbac-proxy
17:53:45	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r	Started	Started container kube-rbac-proxy
17:53:49	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:53:49	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:54:52 (x9)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r	Unhealthy	Readiness probe failed: dial tcp 10.133.0.46:8080: connect: connection refused
17:55:02 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:55:02	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-runtime] is Ready
17:55:04	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r	Killing	Stopping container kube-rbac-proxy
17:55:04 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:55:04 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:55:04	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r	Killing	Stopping container kserve-container
17:55:07	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-p6h5r	Unhealthy	Readiness probe failed: Get "https://10.133.0.46:8643/healthz": dial tcp 10.133.0.46:8643: connect: connection refused
17:55:09	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-runtime-predictor-7f4957c5d5 from 0 to 1
17:55:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm	Created	Created container: storage-initializer
17:55:10 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
17:55:10	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5	SuccessfulCreate	Created pod: isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm
17:55:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm	Started	Started container storage-initializer
17:55:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
17:55:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
17:55:10	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm	AddedInterface	Add eth0 [10.134.0.27/23] from ovn-kubernetes
17:55:14	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm	Pulling	Pulling image "docker.io/seldonio/mlserver:1.7.1"
17:57:23	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm	Created	Created container: kserve-container
17:57:23	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:57:23	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm	Created	Created container: kube-rbac-proxy
17:57:23	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm	Started	Started container kserve-container
17:57:23	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm	Started	Started container kube-rbac-proxy
17:57:23	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm	Pulled	Successfully pulled image "docker.io/seldonio/mlserver:1.7.1" in 2m9.217s (2m9.217s including waiting). Image size: 10890461297 bytes.
17:57:55 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:57:55 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:58:00 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:58:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-runtime] is Ready
17:58:10	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedUpdateStatus	horizontalpodautoscalers.autoscaling "isvc-lightgbm-v2-runtime-predictor" not found
17:58:10	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:58:10	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:58:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm	Killing	Stopping container kube-rbac-proxy
17:58:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jvlcm	Killing	Stopping container kserve-container
17:58:15	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5	SuccessfulCreate	Created pod: isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt
17:58:15	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5 from 0 to 1
17:58:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
17:58:15	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
17:58:16	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt	AddedInterface	Add eth0 [10.134.0.28/23] from ovn-kubernetes
17:58:16	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
17:58:16	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt	Created	Created container: storage-initializer
17:58:16	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt	Started	Started container storage-initializer
17:58:20	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt	Pulled	Container image "kserve/lgbserver:latest" already present on machine
17:58:20	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt	Created	Created container: kserve-container
17:58:20	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt	Started	Started container kserve-container
17:58:20	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:58:20	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt	Created	Created container: kube-rbac-proxy
17:58:20	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt	Started	Started container kube-rbac-proxy
17:58:23 (x2)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt	Unhealthy	Readiness probe failed: dial tcp 10.134.0.28:8080: connect: connection refused
17:58:28 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:58:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-kserve] is Ready
17:58:30	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:58:30	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:58:35	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt	Killing	Stopping container kube-rbac-proxy
17:58:35	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ljwxt	Killing	Stopping container kserve-container
17:58:40	kserve-ci-e2e-test	deployment-controller	isvc-mlflow-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-mlflow-v2-runtime-predictor-6456f94494 from 0 to 1
17:58:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-mlflow-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-mlflow-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
17:58:40	kserve-ci-e2e-test	replicaset-controller	isvc-mlflow-v2-runtime-predictor-6456f94494	SuccessfulCreate	Created pod: isvc-mlflow-v2-runtime-predictor-6456f94494-6z977
17:58:40	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-6z977	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
17:58:41	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-6z977	Created	Created container: storage-initializer
17:58:41	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-6z977	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
17:58:41	kserve-ci-e2e-test	multus	isvc-mlflow-v2-runtime-predictor-6456f94494-6z977	AddedInterface	Add eth0 [10.134.0.29/23] from ovn-kubernetes
17:58:41	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-6z977	Started	Started container storage-initializer
17:58:46	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-6z977	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:58:46	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-6z977	Started	Started container kserve-container
17:58:46	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-6z977	Created	Created container: kube-rbac-proxy
17:58:46	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-6z977	Created	Created container: kserve-container
17:58:46	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-6z977	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
17:58:46	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-6z977	Started	Started container kube-rbac-proxy
17:59:23 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:59:23	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	InferenceServiceReady	InferenceService [isvc-mlflow-v2-runtime] is Ready
17:59:25 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:59:25 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:59:30	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-6z977	Killing	Stopping container kserve-container
17:59:30	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-6z977	Killing	Stopping container kube-rbac-proxy
17:59:35	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-mcp-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-mcp-predictor-5996957568 from 0 to 1
17:59:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-mcp": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-mcp": the object has been modified; please apply your changes to the latest version and try again
17:59:35	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-mcp-predictor-5996957568	SuccessfulCreate	Created pod: isvc-sklearn-mcp-predictor-5996957568-pgdxp
17:59:36	kserve-ci-e2e-test	multus	isvc-sklearn-mcp-predictor-5996957568-pgdxp	AddedInterface	Add eth0 [10.134.0.30/23] from ovn-kubernetes
17:59:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5996957568-pgdxp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
17:59:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5996957568-pgdxp	Created	Created container: storage-initializer
17:59:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5996957568-pgdxp	Started	Started container storage-initializer
17:59:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5996957568-pgdxp	Started	Started container kserve-container
17:59:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5996957568-pgdxp	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine
17:59:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5996957568-pgdxp	Created	Created container: kserve-container
17:59:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5996957568-pgdxp	Pulling	Pulling image "quay.io/opendatahub/kserve-agent:latest"
17:59:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5996957568-pgdxp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:59:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5996957568-pgdxp	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent:latest" in 2.788s (2.788s including waiting). Image size: 239481897 bytes.
17:59:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5996957568-pgdxp	Created	Created container: kserve-agent
17:59:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5996957568-pgdxp	Started	Started container kserve-agent
17:59:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5996957568-pgdxp	Started	Started container kube-rbac-proxy
17:59:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5996957568-pgdxp	Created	Created container: kube-rbac-proxy
18:00:16	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5996957568-pgdxp	Unhealthy	Liveness probe failed: Get "http://10.134.0.30:8080/v1/models/isvc-sklearn-mcp": dial tcp 10.134.0.30:8080: connect: connection refused
18:00:49 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:00:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	InferenceServiceReady	InferenceService [isvc-sklearn-mcp] is Ready
18:00:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5996957568-pgdxp	Killing	Stopping container kube-rbac-proxy
18:00:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5996957568-pgdxp	Killing	Stopping container kserve-container
18:00:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5996957568-pgdxp	Killing	Stopping container kserve-agent
18:01:01	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-shpq6	Created	Created container: storage-initializer
18:01:01	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-shpq6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:01:01	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-shpq6	Started	Started container storage-initializer
18:01:01	kserve-ci-e2e-test	multus	isvc-paddle-predictor-6d5c9fd4f6-shpq6	AddedInterface	Add eth0 [10.133.0.47/23] from ovn-kubernetes
18:01:01	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-predictor-6d5c9fd4f6	SuccessfulCreate	Created pod: isvc-paddle-predictor-6d5c9fd4f6-shpq6
18:01:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	UpdateFailed	Failed to update status for InferenceService "isvc-paddle": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle": the object has been modified; please apply your changes to the latest version and try again
18:01:01	kserve-ci-e2e-test	deployment-controller	isvc-paddle-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-predictor-6d5c9fd4f6 from 0 to 1
18:01:08	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-shpq6	Pulling	Pulling image "kserve/paddleserver:latest"
18:01:19	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-shpq6	Created	Created container: kube-rbac-proxy
18:01:19	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-shpq6	Created	Created container: kserve-container
18:01:19	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-shpq6	Started	Started container kserve-container
18:01:19 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5996957568-pgdxp	Unhealthy	Readiness probe failed: Get "http://10.134.0.30:8080/v1/models/isvc-sklearn-mcp": dial tcp 10.134.0.30:8080: connect: connection refused
18:01:19	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-shpq6	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:01:19	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-shpq6	Pulled	Successfully pulled image "kserve/paddleserver:latest" in 11.551s (11.551s including waiting). Image size: 1163606775 bytes.
18:01:19	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-shpq6	Started	Started container kube-rbac-proxy
18:01:24 (x6)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5996957568-pgdxp	Unhealthy	Readiness probe failed: Get "https://10.134.0.30:8643/healthz": dial tcp 10.134.0.30:8643: connect: connection refused
18:01:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:01:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:01:57 (x6)	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-shpq6	Unhealthy	Readiness probe failed: dial tcp 10.133.0.47:8080: connect: connection refused
18:02:01 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:02:01 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:02:07	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	InferenceServiceReady	InferenceService [isvc-paddle] is Ready
18:02:07 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:02:12	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-shpq6	Killing	Stopping container kube-rbac-proxy
18:02:12	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-shpq6	Killing	Stopping container kserve-container
18:02:17	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-runtime": the object has been modified; please apply your changes to the latest version and try again
18:02:17	kserve-ci-e2e-test	deployment-controller	isvc-paddle-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-runtime-predictor-58b99fff76 from 0 to 1
18:02:18	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-cvnvf	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-paddle-runtime-predictor-serving-cert" not found
18:02:18	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-runtime-predictor-58b99fff76	SuccessfulCreate	Created pod: isvc-paddle-runtime-predictor-58b99fff76-cvnvf
18:02:19	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-cvnvf	Created	Created container: storage-initializer
18:02:19	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-cvnvf	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:02:19	kserve-ci-e2e-test	multus	isvc-paddle-runtime-predictor-58b99fff76-cvnvf	AddedInterface	Add eth0 [10.133.0.48/23] from ovn-kubernetes
18:02:19	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-cvnvf	Started	Started container storage-initializer
18:02:24	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-cvnvf	Created	Created container: kube-rbac-proxy
18:02:24	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-cvnvf	Started	Started container kube-rbac-proxy
18:02:24	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-cvnvf	Pulled	Container image "kserve/paddleserver:latest" already present on machine
18:02:24	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-cvnvf	Created	Created container: kserve-container
18:02:24	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-cvnvf	Started	Started container kserve-container
18:02:24	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-cvnvf	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:02:48 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:02:48 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:03:02 (x6)	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-cvnvf	Unhealthy	Readiness probe failed: dial tcp 10.133.0.48:8080: connect: connection refused
18:03:12 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:03:12	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	InferenceServiceReady	InferenceService [isvc-paddle-runtime] is Ready
18:03:18 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:03:18 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:03:19	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-cvnvf	Killing	Stopping container kube-rbac-proxy
18:03:19	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-cvnvf	Killing	Stopping container kserve-container
18:03:24	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-v2-kserve-predictor-64d4c74b4d	SuccessfulCreate	Created pod: isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5
18:03:24	kserve-ci-e2e-test	deployment-controller	isvc-paddle-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-v2-kserve-predictor-64d4c74b4d from 0 to 1
18:03:24	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
18:03:25	kserve-ci-e2e-test	multus	isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5	AddedInterface	Add eth0 [10.133.0.49/23] from ovn-kubernetes
18:03:25	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:03:25	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5	Created	Created container: storage-initializer
18:03:25	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5	Started	Started container storage-initializer
18:03:29	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5	Created	Created container: kube-rbac-proxy
18:03:29	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5	Started	Started container kserve-container
18:03:29	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5	Started	Started container kube-rbac-proxy
18:03:29	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:03:29	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5	Pulled	Container image "kserve/paddleserver:latest" already present on machine
18:03:29	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5	Created	Created container: kserve-container
18:03:39	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:03:39	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:04:17	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	InferenceServiceReady	InferenceService [isvc-paddle-v2-kserve] is Ready
18:04:17 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:04:24 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:04:24 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:04:26	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5	Killing	Stopping container kserve-container
18:04:26	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5	Killing	Stopping container kube-rbac-proxy
18:04:27 (x7)	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5	Unhealthy	Readiness probe failed: dial tcp 10.133.0.49:8080: connect: connection refused
18:04:27	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-mcgh5	Unhealthy	Readiness probe failed: Get "https://10.133.0.49:8643/healthz": dial tcp 10.133.0.49:8643: connect: connection refused
18:04:31	kserve-ci-e2e-test	deployment-controller	isvc-pmml-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-predictor-8674795f59 from 0 to 1
18:04:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	UpdateFailed	Failed to update status for InferenceService "isvc-pmml": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml": the object has been modified; please apply your changes to the latest version and try again
18:04:32	kserve-ci-e2e-test	multus	isvc-pmml-predictor-8674795f59-tpxjh	AddedInterface	Add eth0 [10.134.0.31/23] from ovn-kubernetes
18:04:32	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-predictor-8674795f59	SuccessfulCreate	Created pod: isvc-pmml-predictor-8674795f59-tpxjh
18:04:32	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tpxjh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:04:32	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tpxjh	Created	Created container: storage-initializer
18:04:32	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tpxjh	Started	Started container storage-initializer
18:04:36	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tpxjh	Pulling	Pulling image "kserve/pmmlserver:latest"
18:04:43	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tpxjh	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:04:43	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tpxjh	Created	Created container: kserve-container
18:04:43	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tpxjh	Started	Started container kserve-container
18:04:43	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tpxjh	Created	Created container: kube-rbac-proxy
18:04:43	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tpxjh	Started	Started container kube-rbac-proxy
18:04:43	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tpxjh	Pulled	Successfully pulled image "kserve/pmmlserver:latest" in 6.784s (6.784s including waiting). Image size: 801937279 bytes.
18:04:46	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:04:47	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:05:57 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tpxjh	Unhealthy	Readiness probe failed: dial tcp 10.134.0.31:8080: connect: connection refused
18:06:02 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:06:02 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:06:07	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	InferenceServiceReady	InferenceService [isvc-pmml] is Ready
18:06:07 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:06:13	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tpxjh	Killing	Stopping container kserve-container
18:06:13	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tpxjh	Killing	Stopping container kube-rbac-proxy
18:06:15	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-tpxjh	Unhealthy	Readiness probe failed: Get "https://10.134.0.31:8643/healthz": dial tcp 10.134.0.31:8643: connect: connection refused
18:06:18	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-87tld	Created	Created container: storage-initializer
18:06:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-runtime": the object has been modified; please apply your changes to the latest version and try again
18:06:18	kserve-ci-e2e-test	deployment-controller	isvc-pmml-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-runtime-predictor-885fc8777 from 0 to 1
18:06:18	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-runtime-predictor-885fc8777	SuccessfulCreate	Created pod: isvc-pmml-runtime-predictor-885fc8777-87tld
18:06:18 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-runtime": the object has been modified; please apply your changes to the latest version and try again
18:06:18	kserve-ci-e2e-test	multus	isvc-pmml-runtime-predictor-885fc8777-87tld	AddedInterface	Add eth0 [10.134.0.32/23] from ovn-kubernetes
18:06:18	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-87tld	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:06:18	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-87tld	Started	Started container storage-initializer
18:06:23	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-87tld	Started	Started container kserve-container
18:06:23	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-87tld	Created	Created container: kube-rbac-proxy
18:06:23	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-87tld	Created	Created container: kserve-container
18:06:23	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-87tld	Started	Started container kube-rbac-proxy
18:06:23	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-87tld	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
18:06:23	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-87tld	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:06:33	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:06:33	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:07:41 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-87tld	Unhealthy	Readiness probe failed: dial tcp 10.134.0.32:8080: connect: connection refused
18:07:44 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:07:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	InferenceServiceReady	InferenceService [isvc-pmml-runtime] is Ready
18:07:48 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:07:48 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:07:49	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-87tld	Killing	Stopping container kube-rbac-proxy
18:07:49	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-87tld	Killing	Stopping container kserve-container
18:07:51	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-87tld	Unhealthy	Readiness probe failed: Get "https://10.134.0.32:8643/healthz": dial tcp 10.134.0.32:8643: connect: connection refused
18:07:54	kserve-ci-e2e-test	deployment-controller	isvc-pmml-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-v2-kserve-predictor-84cb55486c from 0 to 1
18:07:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
18:07:54	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-v2-kserve-predictor-84cb55486c	SuccessfulCreate	Created pod: isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf
18:07:55	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf	Started	Started container storage-initializer
18:07:55	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf	Created	Created container: storage-initializer
18:07:55	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:07:55	kserve-ci-e2e-test	multus	isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf	AddedInterface	Add eth0 [10.134.0.33/23] from ovn-kubernetes
18:07:59	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf	Started	Started container kube-rbac-proxy
18:07:59	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
18:07:59	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf	Created	Created container: kserve-container
18:07:59	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf	Started	Started container kserve-container
18:07:59	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:07:59	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf	Created	Created container: kube-rbac-proxy
18:08:39 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:08:39 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:09:16 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:09:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	InferenceServiceReady	InferenceService [isvc-pmml-v2-kserve] is Ready
18:09:24 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:09:24 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:09:25	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf	Killing	Stopping container kserve-container
18:09:25	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf	Killing	Stopping container kube-rbac-proxy
18:09:26 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf	Unhealthy	Readiness probe failed: dial tcp 10.134.0.33:8080: connect: connection refused
18:09:26	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2zpsf	Unhealthy	Readiness probe failed: Get "https://10.134.0.33:8643/healthz": dial tcp 10.134.0.33:8643: connect: connection refused
18:09:31	kserve-ci-e2e-test	replicaset-controller	isvc-primary-a0e545-predictor-865b97b54f	SuccessfulCreate	Created pod: isvc-primary-a0e545-predictor-865b97b54f-v9n79
18:09:31	kserve-ci-e2e-test	multus	isvc-primary-a0e545-predictor-865b97b54f-v9n79	AddedInterface	Add eth0 [10.134.0.34/23] from ovn-kubernetes
18:09:31	kserve-ci-e2e-test	kubelet	isvc-primary-a0e545-predictor-865b97b54f-v9n79	Created	Created container: storage-initializer
18:09:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-a0e545	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-a0e545": the object has been modified; please apply your changes to the latest version and try again
18:09:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-a0e545	UpdateFailed	Failed to update status for InferenceService "isvc-primary-a0e545": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-a0e545": the object has been modified; please apply your changes to the latest version and try again
18:09:31	kserve-ci-e2e-test	kubelet	isvc-primary-a0e545-predictor-865b97b54f-v9n79	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:09:31	kserve-ci-e2e-test	deployment-controller	isvc-primary-a0e545-predictor	ScalingReplicaSet	Scaled up replica set isvc-primary-a0e545-predictor-865b97b54f from 0 to 1
18:09:31	kserve-ci-e2e-test	kubelet	isvc-primary-a0e545-predictor-865b97b54f-v9n79	Started	Started container storage-initializer
18:09:35	kserve-ci-e2e-test	kubelet	isvc-primary-a0e545-predictor-865b97b54f-v9n79	Created	Created container: kserve-container
18:09:35	kserve-ci-e2e-test	kubelet	isvc-primary-a0e545-predictor-865b97b54f-v9n79	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine
18:09:35	kserve-ci-e2e-test	kubelet	isvc-primary-a0e545-predictor-865b97b54f-v9n79	Started	Started container kube-rbac-proxy
18:09:35	kserve-ci-e2e-test	kubelet	isvc-primary-a0e545-predictor-865b97b54f-v9n79	Created	Created container: kube-rbac-proxy
18:09:35	kserve-ci-e2e-test	kubelet	isvc-primary-a0e545-predictor-865b97b54f-v9n79	Started	Started container kserve-container
18:09:35	kserve-ci-e2e-test	kubelet	isvc-primary-a0e545-predictor-865b97b54f-v9n79	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:10:01 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-a0e545-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:10:01 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-a0e545-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:10:43 (x9)	kserve-ci-e2e-test	kubelet	isvc-primary-a0e545-predictor-865b97b54f-v9n79	Unhealthy	Readiness probe failed: dial tcp 10.134.0.34:8080: connect: connection refused
18:10:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-a0e545	InferenceServiceReady	InferenceService [isvc-primary-a0e545] is Ready
18:10:49 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-a0e545	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:11:01 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-a0e545-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:11:01 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-a0e545-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:11:06 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-a0e545	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-a0e545": the object has been modified; please apply your changes to the latest version and try again
18:11:06	kserve-ci-e2e-test	replicaset-controller	isvc-secondary-a0e545-predictor-74b7c6689d	SuccessfulCreate	Created pod: isvc-secondary-a0e545-predictor-74b7c6689d-wgr69
18:11:06	kserve-ci-e2e-test	deployment-controller	isvc-secondary-a0e545-predictor	ScalingReplicaSet	Scaled up replica set isvc-secondary-a0e545-predictor-74b7c6689d from 0 to 1
18:11:06 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-a0e545	UpdateFailed	Failed to update status for InferenceService "isvc-secondary-a0e545": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-a0e545": the object has been modified; please apply your changes to the latest version and try again
18:11:06	kserve-ci-e2e-test	multus	isvc-secondary-a0e545-predictor-74b7c6689d-wgr69	AddedInterface	Add eth0 [10.133.0.50/23] from ovn-kubernetes
18:11:09 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-a0e545-predictor-74b7c6689d-wgr69	Created	Created container: storage-initializer
18:11:09 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-a0e545-predictor-74b7c6689d-wgr69	Started	Started container storage-initializer
18:11:09 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-a0e545-predictor-74b7c6689d-wgr69	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:11:13 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-a0e545	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:11:13	kserve-ci-e2e-test	kubelet	isvc-secondary-a0e545-predictor-74b7c6689d-wgr69	BackOff	Back-off restarting failed container storage-initializer in pod isvc-secondary-a0e545-predictor-74b7c6689d-wgr69_kserve-ci-e2e-test(ac427654-7353-475f-8476-187e99f10e3e)
18:11:20	kserve-ci-e2e-test	kubelet	isvc-primary-a0e545-predictor-865b97b54f-v9n79	Killing	Stopping container kserve-container
18:11:20	kserve-ci-e2e-test	kubelet	isvc-primary-a0e545-predictor-865b97b54f-v9n79	Killing	Stopping container kube-rbac-proxy
18:11:23	kserve-ci-e2e-test	kubelet	isvc-primary-a0e545-predictor-865b97b54f-v9n79	Unhealthy	Readiness probe failed: Get "https://10.134.0.34:8643/healthz": dial tcp 10.134.0.34:8643: connect: connection refused
18:11:25	kserve-ci-e2e-test	deployment-controller	isvc-init-fail-048456-predictor	ScalingReplicaSet	Scaled up replica set isvc-init-fail-048456-predictor-78ffc8bb9d from 0 to 1
18:11:25	kserve-ci-e2e-test	replicaset-controller	isvc-init-fail-048456-predictor-78ffc8bb9d	SuccessfulCreate	Created pod: isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb
18:11:25	kserve-ci-e2e-test	kubelet	isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-init-fail-048456-predictor-serving-cert" not found
18:11:25 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-048456	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-048456": the object has been modified; please apply your changes to the latest version and try again
18:11:25 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-048456	UpdateFailed	Failed to update status for InferenceService "isvc-init-fail-048456": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-048456": the object has been modified; please apply your changes to the latest version and try again
18:11:26	kserve-ci-e2e-test	multus	isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb	AddedInterface	Add eth0 [10.134.0.35/23] from ovn-kubernetes
18:11:32 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-048456	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:11:32 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb	Started	Started container storage-initializer
18:11:32 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb	Created	Created container: storage-initializer
18:11:32 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:11:35	kserve-ci-e2e-test	kubelet	isvc-init-fail-048456-predictor-78ffc8bb9d-4jnqb	Killing	Stopping container storage-initializer
18:11:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn": the object has been modified; please apply your changes to the latest version and try again
18:11:40	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-predictor-5f849d69df from 0 to 1
18:11:40	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-predictor-5f849d69df	SuccessfulCreate	Created pod: isvc-predictive-sklearn-predictor-5f849d69df-ljhq9
18:11:41	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-predictor-5f849d69df-ljhq9	AddedInterface	Add eth0 [10.133.0.51/23] from ovn-kubernetes
18:11:41	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-ljhq9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:11:41	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-ljhq9	Created	Created container: storage-initializer
18:11:41	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-ljhq9	Started	Started container storage-initializer
18:11:45	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-ljhq9	Pulling	Pulling image "kserve/predictiveserver:latest"
18:12:08	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-ljhq9	Pulled	Successfully pulled image "kserve/predictiveserver:latest" in 22.609s (22.609s including waiting). Image size: 2352711313 bytes.
18:12:08	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-ljhq9	Started	Started container kserve-container
18:12:08	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-ljhq9	Created	Created container: kube-rbac-proxy
18:12:08	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-ljhq9	Started	Started container kube-rbac-proxy
18:12:08	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-ljhq9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:12:08	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-ljhq9	Created	Created container: kserve-container
18:12:40 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:12:40 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:13:17 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-ljhq9	Unhealthy	Readiness probe failed: dial tcp 10.133.0.51:8080: connect: connection refused
18:13:26 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:13:26 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:13:27 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:13:27	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	InferenceServiceReady	InferenceService [isvc-predictive-sklearn] is Ready
18:13:31	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-ljhq9	Killing	Stopping container kube-rbac-proxy
18:13:31	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-ljhq9	Killing	Stopping container kserve-container
18:13:36	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-z2vth	Created	Created container: storage-initializer
18:13:36	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-z2vth	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:13:36	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost": the object has been modified; please apply your changes to the latest version and try again
18:13:36 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost": the object has been modified; please apply your changes to the latest version and try again
18:13:36	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-predictor-695f978978	SuccessfulCreate	Created pod: isvc-predictive-xgboost-predictor-695f978978-z2vth
18:13:36	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-predictor-695f978978 from 0 to 1
18:13:36	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-z2vth	Started	Started container storage-initializer
18:13:36	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-predictor-695f978978-z2vth	AddedInterface	Add eth0 [10.134.0.36/23] from ovn-kubernetes
18:13:41	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-z2vth	Pulling	Pulling image "kserve/predictiveserver:latest"
18:14:02	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-z2vth	Pulled	Successfully pulled image "kserve/predictiveserver:latest" in 21.644s (21.644s including waiting). Image size: 2352711313 bytes.
18:14:03	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-z2vth	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:14:03	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-z2vth	Created	Created container: kube-rbac-proxy
18:14:03	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-z2vth	Created	Created container: kserve-container
18:14:03	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-z2vth	Started	Started container kserve-container
18:14:03	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-z2vth	Started	Started container kube-rbac-proxy
18:14:51 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:14:51 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:15:20 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:15:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	InferenceServiceReady	InferenceService [isvc-predictive-xgboost] is Ready
18:15:21 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:15:21 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:15:26	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-z2vth	Killing	Stopping container kserve-container
18:15:26	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-z2vth	Killing	Stopping container kube-rbac-proxy
18:15:30	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-z2vth	Unhealthy	Readiness probe failed: Get "https://10.134.0.36:8643/healthz": dial tcp 10.134.0.36:8643: connect: connection refused
18:15:30 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-z2vth	Unhealthy	Readiness probe failed: dial tcp 10.134.0.36:8080: connect: connection refused
18:15:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm": the object has been modified; please apply your changes to the latest version and try again
18:15:31	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-predictor-ff56bcbb5 from 0 to 1
18:15:31	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-predictor-ff56bcbb5	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h
18:15:32	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h	Started	Started container storage-initializer
18:15:32	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h	AddedInterface	Add eth0 [10.134.0.37/23] from ovn-kubernetes
18:15:32	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:15:32	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h	Created	Created container: storage-initializer
18:15:36	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h	Created	Created container: kserve-container
18:15:36	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:15:36	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
18:15:36	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h	Started	Started container kserve-container
18:15:36	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h	Started	Started container kube-rbac-proxy
18:15:36	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h	Created	Created container: kube-rbac-proxy
18:16:01 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:16:01 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:16:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm] is Ready
18:16:53 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:17:01	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h	Killing	Stopping container kube-rbac-proxy
18:17:01 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:17:01 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:17:01	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h	Killing	Stopping container kserve-container
18:17:03 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h	Unhealthy	Readiness probe failed: dial tcp 10.134.0.37:8080: connect: connection refused
18:17:03	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-82x7h	Unhealthy	Readiness probe failed: Get "https://10.134.0.37:8643/healthz": dial tcp 10.134.0.37:8643: connect: connection refused
18:17:06	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
18:17:06 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
18:17:06	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-v2-predictor-745f9c8c9f from 0 to 1
18:17:06	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-v2-predictor-745f9c8c9f	SuccessfulCreate	Created pod: isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd
18:17:07	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:17:07	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd	Started	Started container storage-initializer
18:17:07	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd	Created	Created container: storage-initializer
18:17:07	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
18:17:12	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:17:12	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
18:17:12	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd	Started	Started container kserve-container
18:17:12	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd	Created	Created container: kube-rbac-proxy
18:17:12	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd	Started	Started container kube-rbac-proxy
18:17:12	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd	Created	Created container: kserve-container
18:17:36 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:17:36 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:18:28 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:18:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	InferenceServiceReady	InferenceService [isvc-predictive-sklearn-v2] is Ready
18:18:36 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:18:36 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:18:37	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd	Killing	Stopping container kube-rbac-proxy
18:18:37	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd	Killing	Stopping container kserve-container
18:18:38 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd	Unhealthy	Readiness probe failed: Get "http://10.134.0.38:8080/v2/models/isvc-predictive-sklearn-v2/ready": dial tcp 10.134.0.38:8080: connect: connection refused
18:18:40	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-l7nkd	Unhealthy	Readiness probe failed: Get "https://10.134.0.38:8643/healthz": dial tcp 10.134.0.38:8643: connect: connection refused
18:18:42	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-v2-predictor-868ff5dd6d	SuccessfulCreate	Created pod: isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr
18:18:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
18:18:42	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-v2-predictor-868ff5dd6d from 0 to 1
18:18:43	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr	AddedInterface	Add eth0 [10.133.0.52/23] from ovn-kubernetes
18:18:43	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr	Started	Started container storage-initializer
18:18:43	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr	Created	Created container: storage-initializer
18:18:43	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:18:47	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr	Created	Created container: kube-rbac-proxy
18:18:47	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr	Created	Created container: kserve-container
18:18:47	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr	Started	Started container kube-rbac-proxy
18:18:47	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr	Started	Started container kserve-container
18:18:47	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:18:47	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
18:19:27 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:19:27 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:20:05	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	InferenceServiceReady	InferenceService [isvc-predictive-xgboost-v2] is Ready
18:20:05 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:20:13	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr	Killing	Stopping container kube-rbac-proxy
18:20:13 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:20:13	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr	Killing	Stopping container kserve-container
18:20:13 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:20:15	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr	Unhealthy	Readiness probe failed: Get "https://10.133.0.52:8643/healthz": dial tcp 10.133.0.52:8643: connect: connection refused
18:20:15 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rh2jr	Unhealthy	Readiness probe failed: Get "http://10.133.0.52:8080/v2/models/isvc-predictive-xgboost-v2/ready": dial tcp 10.133.0.52:8080: connect: connection refused
18:20:18	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
18:20:18	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:20:18 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm-v2": the object has been modified; please apply your changes to the latest version and try again
18:20:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm-v2": the object has been modified; please apply your changes to the latest version and try again
18:20:18	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl	Started	Started container storage-initializer
18:20:18	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-v2-predictor-545b7cc587 from 0 to 1
18:20:18	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-v2-predictor-545b7cc587	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl
18:20:18	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl	Created	Created container: storage-initializer
18:20:23	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl	Started	Started container kserve-container
18:20:23	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:20:23	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl	Created	Created container: kube-rbac-proxy
18:20:23	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl	Started	Started container kube-rbac-proxy
18:20:23	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl	Created	Created container: kserve-container
18:20:23	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
18:20:33	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:20:33	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:21:41 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:21:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm-v2] is Ready
18:21:48	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl	Killing	Stopping container kserve-container
18:21:48	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl	Killing	Stopping container kube-rbac-proxy
18:21:48 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:21:48 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:21:51 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl	Unhealthy	Readiness probe failed: Get "http://10.134.0.39:8080/v2/models/isvc-predictive-lightgbm-v2/ready": dial tcp 10.134.0.39:8080: connect: connection refused
18:21:51	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-phvgl	Unhealthy	Readiness probe failed: Get "https://10.134.0.39:8643/healthz": dial tcp 10.134.0.39:8643: connect: connection refused
18:21:53	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-scheduler-predictor-76d485d498	SuccessfulCreate	Created pod: isvc-sklearn-scheduler-predictor-76d485d498-8jjls
18:21:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
18:21:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-scheduler": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
18:21:53 (x5)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:21:53	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-scheduler-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-scheduler-predictor-76d485d498 from 0 to 1
18:22:00	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-predictor-5f4d89cc44 from 0 to 1
18:22:00	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-predictor-5f4d89cc44	SuccessfulCreate	Created pod: isvc-sklearn-predictor-5f4d89cc44-qhdm8
18:22:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn": the object has been modified; please apply your changes to the latest version and try again
18:22:00 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn": the object has been modified; please apply your changes to the latest version and try again
18:22:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-5f4d89cc44-qhdm8	Created	Created container: storage-initializer
18:22:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-5f4d89cc44-qhdm8	Started	Started container storage-initializer
18:22:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-5f4d89cc44-qhdm8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:22:01	kserve-ci-e2e-test	multus	isvc-sklearn-predictor-5f4d89cc44-qhdm8	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
18:22:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-5f4d89cc44-qhdm8	Created	Created container: kube-rbac-proxy
18:22:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-5f4d89cc44-qhdm8	Started	Started container kserve-container
18:22:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-5f4d89cc44-qhdm8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:22:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-5f4d89cc44-qhdm8	Created	Created container: kserve-container
18:22:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-5f4d89cc44-qhdm8	Started	Started container kube-rbac-proxy
18:22:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-5f4d89cc44-qhdm8	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine
18:22:30 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:22:30 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:23:18 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-5f4d89cc44-qhdm8	Unhealthy	Readiness probe failed: dial tcp 10.134.0.40:8080: connect: connection refused
18:23:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	InferenceServiceReady	InferenceService [isvc-sklearn] is Ready
18:23:28 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:23:30	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-5f4d89cc44-qhdm8	Killing	Stopping container kube-rbac-proxy
18:23:30	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-5f4d89cc44-qhdm8	Killing	Stopping container kserve-container
18:23:30 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:23:30 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:23:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-5f4d89cc44-qhdm8	Unhealthy	Readiness probe failed: Get "https://10.134.0.40:8643/healthz": dial tcp 10.134.0.40:8643: connect: connection refused
18:23:36	kserve-ci-e2e-test	deployment-controller	sklearn-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set sklearn-v2-mlserver-predictor-7f568cdc6b from 0 to 1
18:23:36	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
18:23:36 (x2)	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "sklearn-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "sklearn-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
18:23:36	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "sklearn-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
18:23:36	kserve-ci-e2e-test	replicaset-controller	sklearn-v2-mlserver-predictor-7f568cdc6b	SuccessfulCreate	Created pod: sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg
18:23:37	kserve-ci-e2e-test	multus	sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg	AddedInterface	Add eth0 [10.134.0.41/23] from ovn-kubernetes
18:23:37	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:23:37	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg	Created	Created container: storage-initializer
18:23:37	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg	Started	Started container storage-initializer
18:23:42	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:23:42	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
18:23:42	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg	Created	Created container: kserve-container
18:23:42	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg	Started	Started container kserve-container
18:23:42	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg	Created	Created container: kube-rbac-proxy
18:23:42	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg	Started	Started container kube-rbac-proxy
18:24:06 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:24:06 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:24:19	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
18:24:29 (x12)	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:24:29	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	InferenceServiceReady	InferenceService [sklearn-v2-mlserver] is Ready
18:24:36	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg	Killing	Stopping container kube-rbac-proxy
18:24:36 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:24:36 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:24:36	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg	Killing	Stopping container kserve-container
18:24:39	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-l4xzg	Unhealthy	Readiness probe failed: Get "https://10.134.0.41:8643/healthz": dial tcp 10.134.0.41:8643: connect: connection refused
18:24:41	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-runtime-predictor-599b58cf6 from 0 to 1
18:24:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-runtime": the object has been modified; please apply your changes to the latest version and try again
18:24:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-599b58cf6-qw8qz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:24:41	kserve-ci-e2e-test	multus	isvc-sklearn-runtime-predictor-599b58cf6-qw8qz	AddedInterface	Add eth0 [10.133.0.53/23] from ovn-kubernetes
18:24:41	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-runtime-predictor-599b58cf6	SuccessfulCreate	Created pod: isvc-sklearn-runtime-predictor-599b58cf6-qw8qz
18:24:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-599b58cf6-qw8qz	Started	Started container storage-initializer
18:24:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-599b58cf6-qw8qz	Created	Created container: storage-initializer
18:24:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-599b58cf6-qw8qz	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine
18:24:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-599b58cf6-qw8qz	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:24:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-599b58cf6-qw8qz	Started	Started container kube-rbac-proxy
18:24:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-599b58cf6-qw8qz	Created	Created container: kube-rbac-proxy
18:24:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-599b58cf6-qw8qz	Started	Started container kserve-container
18:24:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-599b58cf6-qw8qz	Created	Created container: kserve-container
18:24:54 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-599b58cf6-qw8qz	Unhealthy	Readiness probe failed: dial tcp 10.133.0.53:8080: connect: connection refused
18:24:56	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:24:56	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:25:04 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:25:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-runtime] is Ready
18:25:11	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:25:11	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:25:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-599b58cf6-qw8qz	Killing	Stopping container kserve-container
18:25:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-599b58cf6-qw8qz	Killing	Stopping container kube-rbac-proxy
18:25:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-599b58cf6-qw8qz	Unhealthy	Readiness probe failed: Get "https://10.133.0.53:8643/healthz": dial tcp 10.133.0.53:8643: connect: connection refused
18:25:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
18:25:19	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-runtime-predictor-5f69c7d498	SuccessfulCreate	Created pod: isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6
18:25:19	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-runtime-predictor-5f69c7d498 from 0 to 1
18:25:20 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
18:25:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:25:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6	Started	Started container storage-initializer
18:25:21	kserve-ci-e2e-test	multus	isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6	AddedInterface	Add eth0 [10.134.0.42/23] from ovn-kubernetes
18:25:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6	Created	Created container: storage-initializer
18:25:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
18:25:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6	Created	Created container: kube-rbac-proxy
18:25:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6	Created	Created container: kserve-container
18:25:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6	Started	Started container kserve-container
18:25:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6	Started	Started container kube-rbac-proxy
18:25:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:25:49 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:25:49 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:26:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
18:26:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-v2-runtime] is Ready
18:26:14 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:26:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6	Killing	Stopping container kube-rbac-proxy
18:26:19 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:26:19 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:26:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6	Killing	Stopping container kserve-container
18:26:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6	Unhealthy	Readiness probe failed: Get "https://10.134.0.42:8643/healthz": dial tcp 10.134.0.42:8643: connect: connection refused
18:26:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-7fsd6	Unhealthy	Readiness probe failed: Get "http://10.134.0.42:8080/v2/models/isvc-sklearn-v2-runtime/ready": dial tcp 10.134.0.42:8080: connect: connection refused
18:26:24	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-predictor-77f4cf5bd	SuccessfulCreate	Created pod: isvc-sklearn-v2-predictor-77f4cf5bd-z59vd
18:26:24	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-predictor-77f4cf5bd from 0 to 1
18:26:24	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
18:26:25	kserve-ci-e2e-test	multus	isvc-sklearn-v2-predictor-77f4cf5bd-z59vd	AddedInterface	Add eth0 [10.133.0.54/23] from ovn-kubernetes
18:26:25	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-77f4cf5bd-z59vd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:26:25	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-77f4cf5bd-z59vd	Created	Created container: storage-initializer
18:26:25	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-77f4cf5bd-z59vd	Started	Started container storage-initializer
18:26:30	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-77f4cf5bd-z59vd	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine
18:26:30	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-77f4cf5bd-z59vd	Created	Created container: kserve-container
18:26:30	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-77f4cf5bd-z59vd	Started	Started container kserve-container
18:26:30	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-77f4cf5bd-z59vd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:26:30	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-77f4cf5bd-z59vd	Created	Created container: kube-rbac-proxy
18:26:30	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-77f4cf5bd-z59vd	Started	Started container kube-rbac-proxy
18:27:09 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:27:09 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:27:48 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:27:48	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	InferenceServiceReady	InferenceService [isvc-sklearn-v2] is Ready
18:27:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-77f4cf5bd-z59vd	Killing	Stopping container kube-rbac-proxy
18:27:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-77f4cf5bd-z59vd	Killing	Stopping container kserve-container
18:27:55 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:27:55 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:27:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-77f4cf5bd-z59vd	Unhealthy	Readiness probe failed: Get "https://10.133.0.54:8643/healthz": dial tcp 10.133.0.54:8643: connect: connection refused
18:27:58 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-77f4cf5bd-z59vd	Unhealthy	Readiness probe failed: dial tcp 10.133.0.54:8080: connect: connection refused
18:28:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb	Started	Started container storage-initializer
18:28:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-mixed": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-mixed": the object has been modified; please apply your changes to the latest version and try again
18:28:00	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-mixed-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-mixed-predictor-5746969457 from 0 to 1
18:28:00	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-mixed-predictor-5746969457	SuccessfulCreate	Created pod: isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb
18:28:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:28:00	kserve-ci-e2e-test	multus	isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb	AddedInterface	Add eth0 [10.134.0.43/23] from ovn-kubernetes
18:28:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb	Created	Created container: storage-initializer
18:28:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb	Created	Created container: kube-rbac-proxy
18:28:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb	Created	Created container: kserve-container
18:28:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine
18:28:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb	Started	Started container kserve-container
18:28:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:28:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb	Started	Started container kube-rbac-proxy
18:28:30 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:28:30 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:29:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	InferenceServiceReady	InferenceService [isvc-sklearn-v2-mixed] is Ready
18:29:22 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:29:30 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:29:30 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:29:30	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb	Killing	Stopping container kube-rbac-proxy
18:29:30	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb	Killing	Stopping container kserve-container
18:29:32 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb	Unhealthy	Readiness probe failed: dial tcp 10.134.0.43:8080: connect: connection refused
18:29:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5746969457-jjxdb	Unhealthy	Readiness probe failed: Get "https://10.134.0.43:8643/healthz": dial tcp 10.134.0.43:8643: connect: connection refused
18:29:35	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-predictor-547dfc5669	SuccessfulCreate	Created pod: isvc-tensorflow-predictor-547dfc5669-stxfr
18:29:35	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-predictor-547dfc5669 from 0 to 1
18:29:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow": the object has been modified; please apply your changes to the latest version and try again
18:29:36	kserve-ci-e2e-test	multus	isvc-tensorflow-predictor-547dfc5669-stxfr	AddedInterface	Add eth0 [10.134.0.44/23] from ovn-kubernetes
18:29:36	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxfr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:29:36	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxfr	Created	Created container: storage-initializer
18:29:36	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxfr	Started	Started container storage-initializer
18:29:40	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxfr	Pulling	Pulling image "tensorflow/serving:2.6.2"
18:29:44	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxfr	Started	Started container kserve-container
18:29:44	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxfr	Created	Created container: kserve-container
18:29:44	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxfr	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:29:44	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxfr	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 3.676s (3.676s including waiting). Image size: 425873876 bytes.
18:29:44	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxfr	Started	Started container kube-rbac-proxy
18:29:44	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxfr	Created	Created container: kube-rbac-proxy
18:29:50 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxfr	Unhealthy	Readiness probe failed: dial tcp 10.134.0.44:8080: connect: connection refused
18:30:00 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:30:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	InferenceServiceReady	InferenceService [isvc-tensorflow] is Ready
18:30:05 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:30:05 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:30:17	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxfr	Killing	Stopping container kube-rbac-proxy
18:30:17	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxfr	Killing	Stopping container kserve-container
18:30:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow-runtime": the object has been modified; please apply your changes to the latest version and try again
18:30:22	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-runtime-predictor-855cfcd677 from 0 to 1
18:30:22	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
18:30:22	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-runtime-predictor-855cfcd677	SuccessfulCreate	Created pod: isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r
18:30:23	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:30:23	kserve-ci-e2e-test	multus	isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r	AddedInterface	Add eth0 [10.133.0.55/23] from ovn-kubernetes
18:30:23	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r	Created	Created container: storage-initializer
18:30:23	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r	Started	Started container storage-initializer
18:30:28	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r	Pulling	Pulling image "tensorflow/serving:2.6.2"
18:30:31	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 3.736s (3.736s including waiting). Image size: 425873876 bytes.
18:30:31	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r	Created	Created container: kube-rbac-proxy
18:30:31	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r	Started	Started container kube-rbac-proxy
18:30:31	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r	Started	Started container kserve-container
18:30:31	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:30:31	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r	Created	Created container: kserve-container
18:30:37	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:30:37	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:30:38 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r	Unhealthy	Readiness probe failed: dial tcp 10.133.0.55:8080: connect: connection refused
18:30:45 (x6)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxfr	Unhealthy	Readiness probe failed: Get "https://10.134.0.44:8643/healthz": dial tcp 10.134.0.44:8643: connect: connection refused
18:30:48 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:30:48	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	InferenceServiceReady	InferenceService [isvc-tensorflow-runtime] is Ready
18:30:52	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:30:52	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:31:03	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r	Killing	Stopping container kserve-container
18:31:03	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r	Killing	Stopping container kube-rbac-proxy
18:31:08	kserve-ci-e2e-test	deployment-controller	isvc-triton-predictor	ScalingReplicaSet	Scaled up replica set isvc-triton-predictor-5d7db58676 from 0 to 1
18:31:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	UpdateFailed	Failed to update status for InferenceService "isvc-triton": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-triton": the object has been modified; please apply your changes to the latest version and try again
18:31:08	kserve-ci-e2e-test	replicaset-controller	isvc-triton-predictor-5d7db58676	SuccessfulCreate	Created pod: isvc-triton-predictor-5d7db58676-795zp
18:31:09	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-795zp	Started	Started container storage-initializer
18:31:09	kserve-ci-e2e-test	multus	isvc-triton-predictor-5d7db58676-795zp	AddedInterface	Add eth0 [10.134.0.45/23] from ovn-kubernetes
18:31:09	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-795zp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:31:09	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-795zp	Created	Created container: storage-initializer
18:31:12	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-795zp	Pulling	Pulling image "nvcr.io/nvidia/tritonserver:23.05-py3"
18:31:33 (x6)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-ltc5r	Unhealthy	Readiness probe failed: Get "https://10.133.0.55:8643/healthz": dial tcp 10.133.0.55:8643: connect: connection refused
18:33:06	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-795zp	Pulled	Successfully pulled image "nvcr.io/nvidia/tritonserver:23.05-py3" in 1m53.972s (1m53.972s including waiting). Image size: 12907074623 bytes.
18:33:07	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-795zp	Created	Created container: kserve-container
18:33:07	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-795zp	Started	Started container kserve-container
18:33:07	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-795zp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:33:07	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-795zp	Created	Created container: kube-rbac-proxy
18:33:07	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-795zp	Started	Started container kube-rbac-proxy
18:33:08 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:33:08 (x2)	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-795zp	Unhealthy	Readiness probe failed: dial tcp 10.134.0.45:8080: connect: connection refused
18:33:08 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:33:13	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	InferenceServiceReady	InferenceService [isvc-triton] is Ready
18:33:13 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:33:20	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-795zp	Killing	Stopping container kserve-container
18:33:20	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-795zp	Killing	Stopping container kube-rbac-proxy
18:33:25	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-wm285	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:33:25	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-predictor-7566d65bf7 from 0 to 1
18:33:25	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-predictor-7566d65bf7	SuccessfulCreate	Created pod: isvc-xgboost-predictor-7566d65bf7-wm285
18:33:25	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost": the object has been modified; please apply your changes to the latest version and try again
18:33:25	kserve-ci-e2e-test	multus	isvc-xgboost-predictor-7566d65bf7-wm285	AddedInterface	Add eth0 [10.134.0.46/23] from ovn-kubernetes
18:33:25	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-wm285	Started	Started container storage-initializer
18:33:25	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-wm285	Created	Created container: storage-initializer
18:33:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-wm285	Pulling	Pulling image "kserve/xgbserver:latest"
18:33:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-wm285	Started	Started container kube-rbac-proxy
18:33:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-wm285	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 19.845s (19.845s including waiting). Image size: 1307355899 bytes.
18:33:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-wm285	Created	Created container: kserve-container
18:33:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-wm285	Started	Started container kserve-container
18:33:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-wm285	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:33:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-wm285	Created	Created container: kube-rbac-proxy
18:34:25 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:34:25 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:34:55 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:34:55 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:34:57	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	InferenceServiceReady	InferenceService [isvc-xgboost] is Ready
18:34:58 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:35:05	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-wm285	Killing	Stopping container kube-rbac-proxy
18:35:05	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-wm285	Killing	Stopping container kserve-container
18:35:07	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-wm285	Unhealthy	Readiness probe failed: Get "https://10.134.0.46:8643/healthz": dial tcp 10.134.0.46:8643: connect: connection refused
18:35:07 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-wm285	Unhealthy	Readiness probe failed: dial tcp 10.134.0.46:8080: connect: connection refused
18:35:10	kserve-ci-e2e-test	multus	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5	AddedInterface	Add eth0 [10.134.0.47/23] from ovn-kubernetes
18:35:10	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:35:10	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-mlserver-predictor-7b55b84f54	SuccessfulCreate	Created pod: isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5
18:35:10	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-mlserver-predictor-7b55b84f54 from 0 to 1
18:35:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
18:35:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5	Created	Created container: storage-initializer
18:35:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5	Started	Started container storage-initializer
18:35:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5	Created	Created container: kube-rbac-proxy
18:35:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
18:35:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5	Created	Created container: kserve-container
18:35:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5	Started	Started container kserve-container
18:35:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:35:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5	Started	Started container kube-rbac-proxy
18:35:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	InferenceServiceReady	InferenceService [isvc-xgboost-v2-mlserver] is Ready
18:35:53 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:35:55 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:35:55 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:36:00	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5	Killing	Stopping container kube-rbac-proxy
18:36:00	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5	Killing	Stopping container kserve-container
18:36:03	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5	Unhealthy	Readiness probe failed: Get "http://10.134.0.47:8080/v2/models/isvc-xgboost-v2-mlserver/ready": dial tcp 10.134.0.47:8080: connect: connection refused
18:36:03	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zd4n5	Unhealthy	Readiness probe failed: Get "https://10.134.0.47:8643/healthz": dial tcp 10.134.0.47:8643: connect: connection refused
18:36:05	kserve-ci-e2e-test	replicaset-controller	xgboost-v2-mlserver-predictor-59944bcf4c	SuccessfulCreate	Created pod: xgboost-v2-mlserver-predictor-59944bcf4c-xq85r
18:36:05	kserve-ci-e2e-test	deployment-controller	xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set xgboost-v2-mlserver-predictor-59944bcf4c from 0 to 1
18:36:05	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
18:36:06	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-xq85r	Started	Started container storage-initializer
18:36:06	kserve-ci-e2e-test	multus	xgboost-v2-mlserver-predictor-59944bcf4c-xq85r	AddedInterface	Add eth0 [10.134.0.48/23] from ovn-kubernetes
18:36:06	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-xq85r	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:36:06	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-xq85r	Created	Created container: storage-initializer
18:36:10	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-xq85r	Created	Created container: kserve-container
18:36:10	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-xq85r	Started	Started container kube-rbac-proxy
18:36:10	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-xq85r	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
18:36:10	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-xq85r	Started	Started container kserve-container
18:36:10	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-xq85r	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:36:10	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-xq85r	Created	Created container: kube-rbac-proxy
18:36:35	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:36:35	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:36:48	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	InferenceServiceReady	InferenceService [xgboost-v2-mlserver] is Ready
18:36:48 (x11)	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:36:50 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:36:50 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:36:55	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-xq85r	Killing	Stopping container kube-rbac-proxy
18:36:55	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-xq85r	Killing	Stopping container kserve-container
18:36:58	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-xq85r	Unhealthy	Readiness probe failed: Get "https://10.134.0.48:8643/healthz": dial tcp 10.134.0.48:8643: connect: connection refused
18:37:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-runtime": the object has been modified; please apply your changes to the latest version and try again
18:37:00	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-runtime-predictor-54dd488c85 from 0 to 1
18:37:01 (x2)	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-fhs64	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
18:37:01	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-runtime-predictor-54dd488c85	SuccessfulCreate	Created pod: isvc-xgboost-runtime-predictor-54dd488c85-fhs64
18:37:02	kserve-ci-e2e-test	multus	isvc-xgboost-runtime-predictor-54dd488c85-fhs64	AddedInterface	Add eth0 [10.133.0.56/23] from ovn-kubernetes
18:37:03	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-fhs64	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:37:03	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-fhs64	Created	Created container: storage-initializer
18:37:03	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-fhs64	Started	Started container storage-initializer
18:37:07	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-fhs64	Pulling	Pulling image "kserve/xgbserver:latest"
18:37:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-fhs64	Started	Started container kube-rbac-proxy
18:37:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-fhs64	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 18.67s (18.67s including waiting). Image size: 1307355899 bytes.
18:37:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-fhs64	Created	Created container: kserve-container
18:37:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-fhs64	Started	Started container kserve-container
18:37:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-fhs64	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:37:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-fhs64	Created	Created container: kube-rbac-proxy
18:37:46 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:37:46 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:38:31 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:38:31 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:38:33	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-runtime] is Ready
18:38:33 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:38:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-fhs64	Killing	Stopping container kserve-container
18:38:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-fhs64	Killing	Stopping container kube-rbac-proxy
18:38:43 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-fhs64	Unhealthy	Readiness probe failed: dial tcp 10.133.0.56:8080: connect: connection refused
18:38:43	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-fhs64	Unhealthy	Readiness probe failed: Get "https://10.133.0.56:8643/healthz": dial tcp 10.133.0.56:8643: connect: connection refused
18:38:46 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
18:38:46	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
18:38:46	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-runtime-predictor-7787c55dc9	SuccessfulCreate	Created pod: isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt
18:38:46	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-runtime-predictor-7787c55dc9 from 0 to 1
18:38:46	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
18:38:47	kserve-ci-e2e-test	multus	isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt	AddedInterface	Add eth0 [10.134.0.49/23] from ovn-kubernetes
18:38:47	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt	Started	Started container storage-initializer
18:38:47	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt	Created	Created container: storage-initializer
18:38:47	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:38:52	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
18:38:52	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt	Created	Created container: kserve-container
18:38:52	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt	Started	Started container kserve-container
18:38:52	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:38:52	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt	Created	Created container: kube-rbac-proxy
18:38:52	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt	Started	Started container kube-rbac-proxy
18:39:01	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:39:01	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:39:30	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
18:39:40 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:39:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-v2-runtime] is Ready
18:39:46	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt	Killing	Stopping container kserve-container
18:39:46 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:39:46	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt	Killing	Stopping container kube-rbac-proxy
18:39:46 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:39:50	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt	Unhealthy	Readiness probe failed: Get "https://10.134.0.49:8643/healthz": dial tcp 10.134.0.49:8643: connect: connection refused
18:39:50	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-h9tqt	Unhealthy	Readiness probe failed: Get "http://10.134.0.49:8080/v2/models/isvc-xgboost-v2-runtime/ready": dial tcp 10.134.0.49:8080: connect: connection refused
18:39:51	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-predictor-fcc668bc from 0 to 1
18:39:51	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-84dcd	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-v2-predictor-serving-cert" not found
18:39:51	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
18:39:51	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-predictor-fcc668bc	SuccessfulCreate	Created pod: isvc-xgboost-v2-predictor-fcc668bc-84dcd
18:39:52	kserve-ci-e2e-test	multus	isvc-xgboost-v2-predictor-fcc668bc-84dcd	AddedInterface	Add eth0 [10.133.0.57/23] from ovn-kubernetes
18:39:52	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-84dcd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:39:52	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-84dcd	Created	Created container: storage-initializer
18:39:52	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-84dcd	Started	Started container storage-initializer
18:39:57	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-84dcd	Started	Started container kube-rbac-proxy
18:39:57	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-84dcd	Pulled	Container image "kserve/xgbserver:latest" already present on machine
18:39:57	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-84dcd	Created	Created container: kserve-container
18:39:57	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-84dcd	Started	Started container kserve-container
18:39:57	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-84dcd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:39:57	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-84dcd	Created	Created container: kube-rbac-proxy
18:40:21 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:40:21 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:41:04 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:41:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	InferenceServiceReady	InferenceService [isvc-xgboost-v2] is Ready
18:41:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:41:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:41:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-84dcd	Killing	Stopping container kube-rbac-proxy
18:41:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-84dcd	Killing	Stopping container kserve-container
18:41:14	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-84dcd	Unhealthy	Readiness probe failed: Get "https://10.133.0.57:8643/healthz": dial tcp 10.133.0.57:8643: connect: connection refused
18:41:14 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-84dcd	Unhealthy	Readiness probe failed: dial tcp 10.133.0.57:8080: connect: connection refused
18:41:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3": the object has been modified; please apply your changes to the latest version and try again
18:41:16	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-predictor-dd446d6b9	SuccessfulCreate	Created pod: isvc-sklearn-s3-predictor-dd446d6b9-cl7js
18:41:16	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-predictor-dd446d6b9 from 0 to 1
18:41:16 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3": the object has been modified; please apply your changes to the latest version and try again
18:41:17	kserve-ci-e2e-test	multus	isvc-sklearn-s3-predictor-dd446d6b9-cl7js	AddedInterface	Add eth0 [10.133.0.58/23] from ovn-kubernetes
18:41:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-dd446d6b9-cl7js	Created	Created container: storage-initializer
18:41:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-dd446d6b9-cl7js	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:41:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-dd446d6b9-cl7js	Started	Started container storage-initializer
18:41:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-dd446d6b9-cl7js	Created	Created container: kserve-container
18:41:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-dd446d6b9-cl7js	Started	Started container kube-rbac-proxy
18:41:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-dd446d6b9-cl7js	Created	Created container: kube-rbac-proxy
18:41:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-dd446d6b9-cl7js	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:41:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-dd446d6b9-cl7js	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine
18:41:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-dd446d6b9-cl7js	Started	Started container kserve-container
18:41:46 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:41:46 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:42:26 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-dd446d6b9-cl7js	Unhealthy	Readiness probe failed: dial tcp 10.133.0.58:8080: connect: connection refused
18:42:31 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:42:31 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:42:33	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	InferenceServiceReady	InferenceService [isvc-sklearn-s3] is Ready
18:42:33 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:42:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-dd446d6b9-cl7js	Killing	Stopping container kube-rbac-proxy
18:42:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-dd446d6b9-cl7js	Killing	Stopping container kserve-container
18:42:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658	Created	Created container: storage-initializer
18:42:37	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-pass-predictor-665d8df985 from 0 to 1
18:42:37 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-pass": the object has been modified; please apply your changes to the latest version and try again
18:42:37	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-pass": the object has been modified; please apply your changes to the latest version and try again
18:42:37	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-pass-predictor-665d8df985	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658
18:42:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658	Started	Started container storage-initializer
18:42:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:42:37	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658	AddedInterface	Add eth0 [10.133.0.59/23] from ovn-kubernetes
18:42:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine
18:42:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658	Started	Started container kube-rbac-proxy
18:42:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658	Created	Created container: kube-rbac-proxy
18:42:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:42:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658	Started	Started container kserve-container
18:42:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658	Created	Created container: kserve-container
18:42:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-dd446d6b9-cl7js	Unhealthy	Readiness probe failed: Get "https://10.133.0.58:8643/healthz": dial tcp 10.133.0.58:8643: connect: connection refused
18:42:52	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:42:52	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:43:47 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658	Unhealthy	Readiness probe failed: dial tcp 10.133.0.59:8080: connect: connection refused
18:43:57 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:43:57	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-global-pass] is Ready
18:44:07 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:44:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658	Killing	Stopping container kserve-container
18:44:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-ct658	Killing	Stopping container kube-rbac-proxy
18:44:07 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:44:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-fail": the object has been modified; please apply your changes to the latest version and try again
18:44:08	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc from 0 to 1
18:44:08	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5
18:44:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
18:44:09	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5	AddedInterface	Add eth0 [10.134.0.50/23] from ovn-kubernetes
18:44:14 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5	Started	Started container storage-initializer
18:44:14 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:44:14 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5	Created	Created container: storage-initializer
18:44:14 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:44:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-hpbq5	Killing	Stopping container storage-initializer
18:44:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-pass": the object has been modified; please apply your changes to the latest version and try again
18:44:19	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57 from 0 to 1
18:44:19	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55
18:44:19	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55	AddedInterface	Add eth0 [10.133.0.60/23] from ovn-kubernetes
18:44:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:44:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55	Created	Created container: storage-initializer
18:44:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55	Started	Started container storage-initializer
18:44:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine
18:44:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55	Created	Created container: kserve-container
18:44:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:44:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55	Created	Created container: kube-rbac-proxy
18:44:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55	Started	Started container kube-rbac-proxy
18:44:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55	Started	Started container kserve-container
18:44:34	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:44:34	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:45:29 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55	Unhealthy	Readiness probe failed: dial tcp 10.133.0.60:8080: connect: connection refused
18:45:34 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:45:34 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:45:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-custom-pass] is Ready
18:45:39 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:45:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55	Killing	Stopping container kube-rbac-proxy
18:45:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-26m55	Killing	Stopping container kserve-container
18:45:50	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69 from 0 to 1
18:45:50	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2
18:45:50 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-fail": the object has been modified; please apply your changes to the latest version and try again
18:45:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-fail": the object has been modified; please apply your changes to the latest version and try again
18:45:51	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2	AddedInterface	Add eth0 [10.133.0.61/23] from ovn-kubernetes
18:45:56 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:45:56 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2	Created	Created container: storage-initializer
18:45:56 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2	Started	Started container storage-initializer
18:45:56 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:46:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-d7lx2	Killing	Stopping container storage-initializer
18:46:01	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7
18:46:01	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b from 0 to 1
18:46:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
18:46:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-pass": the object has been modified; please apply your changes to the latest version and try again
18:46:01 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-pass": the object has been modified; please apply your changes to the latest version and try again
18:46:02	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7	AddedInterface	Add eth0 [10.133.0.62/23] from ovn-kubernetes
18:46:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:46:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7	Started	Started container storage-initializer
18:46:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7	Created	Created container: storage-initializer
18:46:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:46:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7	Created	Created container: kube-rbac-proxy
18:46:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7	Started	Started container kube-rbac-proxy
18:46:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7	Started	Started container kserve-container
18:46:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7	Created	Created container: kserve-container
18:46:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine
18:46:46 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:46:46 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:47:10 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7	Unhealthy	Readiness probe failed: dial tcp 10.133.0.62:8080: connect: connection refused
18:47:16 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:47:16 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:47:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-serving-pass] is Ready
18:47:21 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:47:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7	Killing	Stopping container kserve-container
18:47:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7	Killing	Stopping container kube-rbac-proxy
18:47:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-fail": the object has been modified; please apply your changes to the latest version and try again
18:47:22	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44 from 0 to 1
18:47:22	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf
18:47:23	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf	AddedInterface	Add eth0 [10.133.0.63/23] from ovn-kubernetes
18:47:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
18:47:25	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-fvgz7	Unhealthy	Readiness probe failed: Get "https://10.133.0.62:8643/healthz": dial tcp 10.133.0.62:8643: connect: connection refused
18:47:26 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf	Started	Started container storage-initializer
18:47:26 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf	Created	Created container: storage-initializer
18:47:26 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine
18:47:32 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:47:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf	BackOff	Back-off restarting failed container storage-initializer in pod isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-5jckf_kserve-ci-e2e-test(27a036ee-1eb6-479c-b69a-80aec8f1312e)