Events

Time	Namespace	Component	RelatedObject	Reason	Message
20:13:17	kserve-ci-e2e-test		isvc-canary-stop-predictor-5d5b899946-lvdqd	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd to ip-10-0-136-207.ec2.internal
19:57:08	kserve-ci-e2e-test		isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz to ip-10-0-136-207.ec2.internal
20:26:21	kserve-ci-e2e-test		isvc-pmml-predictor-8674795f59-6tpkp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp to ip-10-0-136-207.ec2.internal
20:08:31	kserve-ci-e2e-test		isvc-tensorflow-predictor-547dfc5669-kf5wp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-kf5wp to ip-10-0-142-19.ec2.internal
20:25:14	kserve-ci-e2e-test		isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt to ip-10-0-136-207.ec2.internal
20:28:07	kserve-ci-e2e-test		isvc-pmml-runtime-predictor-885fc8777-hvx2w	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w to ip-10-0-136-207.ec2.internal
20:05:20	kserve-ci-e2e-test		isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt to ip-10-0-142-19.ec2.internal
20:27:03	kserve-ci-e2e-test		isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz to ip-10-0-136-207.ec2.internal
20:06:56	kserve-ci-e2e-test		isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8 to ip-10-0-136-207.ec2.internal
20:25:30	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm to ip-10-0-136-207.ec2.internal
20:12:22	kserve-ci-e2e-test		isvc-xgboost-predictor-7566d65bf7-ptfg8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8 to ip-10-0-136-207.ec2.internal
20:09:18	kserve-ci-e2e-test		isvc-tensorflow-runtime-predictor-855cfcd677-cshjb	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb to ip-10-0-136-207.ec2.internal
20:24:07	kserve-ci-e2e-test		isvc-paddle-runtime-predictor-58b99fff76-2xpx4	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4 to ip-10-0-136-207.ec2.internal
20:22:16	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl to ip-10-0-136-207.ec2.internal
20:23:37	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg to ip-10-0-136-207.ec2.internal
20:23:48	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb to ip-10-0-136-207.ec2.internal
20:25:19	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx to ip-10-0-136-207.ec2.internal
20:20:46	kserve-ci-e2e-test		isvc-sklearn-s3-predictor-6c7db45b94-xfs6b	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b to ip-10-0-136-207.ec2.internal
19:51:02	kserve-ci-e2e-test		isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6 to ip-10-0-136-207.ec2.internal
19:54:53	kserve-ci-e2e-test		isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx to ip-10-0-142-19.ec2.internal
20:09:12	kserve-ci-e2e-test		isvc-canary-rollback-v2-predictor-6946b55967-m9f2h	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h to ip-10-0-136-207.ec2.internal
19:50:50	kserve-ci-e2e-test		isvc-logger-predictor-85d5cb84f6-57qgh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh to ip-10-0-136-207.ec2.internal
20:06:06	kserve-ci-e2e-test		isvc-canary-promote-predictor-cb75746f8-74sqd	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd to ip-10-0-136-207.ec2.internal
20:03:02	kserve-ci-e2e-test		isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq to ip-10-0-142-19.ec2.internal
20:17:03	kserve-ci-e2e-test		isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55 to ip-10-0-136-207.ec2.internal
20:17:45	kserve-ci-e2e-test		isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8 to ip-10-0-136-207.ec2.internal
20:19:55	kserve-ci-e2e-test		isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4 to ip-10-0-136-207.ec2.internal
20:22:51	kserve-ci-e2e-test		isvc-paddle-predictor-6d5c9fd4f6-5c6v8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8 to ip-10-0-136-207.ec2.internal
20:18:50	kserve-ci-e2e-test		isvc-xgboost-v2-predictor-fcc668bc-hm8pd	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-hm8pd to ip-10-0-142-19.ec2.internal
20:04:15	kserve-ci-e2e-test		isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5 to ip-10-0-136-207.ec2.internal
20:00:23	kserve-ci-e2e-test		sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6	Scheduled	Successfully assigned kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6 to ip-10-0-136-207.ec2.internal
20:14:58	kserve-ci-e2e-test		isvc-lightgbm-predictor-54f85ddf88-nzvr5	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nzvr5 to ip-10-0-142-19.ec2.internal
20:10:05	kserve-ci-e2e-test		isvc-triton-predictor-5d7db58676-zz2pj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zz2pj to ip-10-0-142-19.ec2.internal
20:13:17	kserve-ci-e2e-test		isvc-canary-stop-v2-predictor-5f87c564d7-64b29	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-64b29 to ip-10-0-142-19.ec2.internal
19:58:47	kserve-ci-e2e-test		isvc-sklearn-predictor-c6b88b959-rbmhx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx to ip-10-0-136-207.ec2.internal
20:30:04	kserve-ci-e2e-test		isvc-primary-274bef-predictor-57b4586b86-zd9wv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv to ip-10-0-136-207.ec2.internal
20:13:17	kserve-ci-e2e-test		isvc-canary-stop-predictor-5d5b899946-9xkt2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-9xkt2 to ip-10-0-142-19.ec2.internal
20:04:31	kserve-ci-e2e-test		isvc-canary-create-predictor-68984b9498-nmsxm	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-nmsxm to ip-10-0-142-19.ec2.internal
19:50:30	kserve-ci-e2e-test		message-dumper-predictor-6575bcfdfd-65ngk	Scheduled	Successfully assigned kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk to ip-10-0-136-207.ec2.internal
20:06:06	kserve-ci-e2e-test		isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl to ip-10-0-136-207.ec2.internal
19:47:00	kserve-ci-e2e-test		isvc-sklearn-batcher-predictor-6cf87485dc-blj8p	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p to ip-10-0-136-207.ec2.internal
20:10:52	kserve-ci-e2e-test		isvc-canary-rollback-predictor-585bd9bcc8-l6xcg	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l6xcg to ip-10-0-142-19.ec2.internal
19:56:28	kserve-ci-e2e-test		isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp to ip-10-0-142-19.ec2.internal
20:20:30	kserve-ci-e2e-test		isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf to ip-10-0-136-207.ec2.internal
20:07:37	kserve-ci-e2e-test		isvc-canary-promote-v2-predictor-c6f7b798f-sn66z	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-sn66z to ip-10-0-142-19.ec2.internal
20:21:26	kserve-ci-e2e-test		isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn to ip-10-0-136-207.ec2.internal
20:06:07	kserve-ci-e2e-test		isvc-canary-promote-predictor-cb75746f8-5lwjh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-5lwjh to ip-10-0-142-19.ec2.internal
19:52:26	kserve-ci-e2e-test		isvc-autogluon-v1-predictor-574b5b7495-df69n	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n to ip-10-0-136-207.ec2.internal
19:54:57	kserve-ci-e2e-test		isvc-autogluon-v2-predictor-9684b4cd9-thssc	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc to ip-10-0-136-207.ec2.internal
20:04:31	kserve-ci-e2e-test		isvc-canary-create-v2-predictor-579784d7d7-fjw2s	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s to ip-10-0-136-207.ec2.internal
19:49:26	kserve-ci-e2e-test		isvc-predictive-xgboost-predictor-695f978978-whrdt	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt to ip-10-0-136-207.ec2.internal
20:04:31	kserve-ci-e2e-test		isvc-canary-create-predictor-68984b9498-sf9gn	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn to ip-10-0-136-207.ec2.internal
19:47:20	kserve-ci-e2e-test		isvc-predictive-sklearn-predictor-5f849d69df-6f5mk	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk to ip-10-0-136-207.ec2.internal
19:52:48	kserve-ci-e2e-test		isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz to ip-10-0-142-19.ec2.internal
20:03:38	kserve-ci-e2e-test		isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz to ip-10-0-136-207.ec2.internal
20:19:00	kserve-ci-e2e-test		isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t to ip-10-0-136-207.ec2.internal
20:16:09	kserve-ci-e2e-test		isvc-xgboost-runtime-predictor-54dd488c85-h8hwc	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc to ip-10-0-136-207.ec2.internal
20:26:52	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n to ip-10-0-136-207.ec2.internal
19:47:00	kserve-ci-e2e-test		isvc-init-fail-c0d673-predictor-6df449b465-6snbz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz to ip-10-0-136-207.ec2.internal
20:14:18	kserve-ci-e2e-test		isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp to ip-10-0-136-207.ec2.internal
20:09:12	kserve-ci-e2e-test		isvc-canary-rollback-predictor-585bd9bcc8-88wtv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-88wtv to ip-10-0-142-19.ec2.internal
19:59:10	kserve-ci-e2e-test		isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5 to ip-10-0-142-19.ec2.internal
20:31:50	kserve-ci-e2e-test		isvc-secondary-274bef-predictor-6665d79445-nw7q9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9 to ip-10-0-136-207.ec2.internal
20:15:13	kserve-ci-e2e-test		xgboost-v2-mlserver-predictor-59944bcf4c-xtj59	Scheduled	Successfully assigned kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59 to ip-10-0-136-207.ec2.internal
20:03:51	kserve-ci-e2e-test		isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl to ip-10-0-142-19.ec2.internal
20:09:12	kserve-ci-e2e-test		isvc-canary-rollback-predictor-585bd9bcc8-6k589	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589 to ip-10-0-136-207.ec2.internal
19:48:55	kserve-ci-e2e-test		isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2 to ip-10-0-136-207.ec2.internal
19:47:00	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-predictor-6cf87485dc	SuccessfulCreate	Created pod: isvc-sklearn-batcher-predictor-6cf87485dc-blj8p
19:47:00	kserve-ci-e2e-test	replicaset-controller	isvc-init-fail-c0d673-predictor-6df449b465	SuccessfulCreate	Created pod: isvc-init-fail-c0d673-predictor-6df449b465-6snbz
19:47:00	kserve-ci-e2e-test	deployment-controller	isvc-init-fail-c0d673-predictor	ScalingReplicaSet	Scaled up replica set isvc-init-fail-c0d673-predictor-6df449b465 from 0 to 1
19:47:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
19:47:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
19:47:00	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-predictor-6cf87485dc from 0 to 1
19:47:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-blj8p	Started	Started container storage-initializer
19:47:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-blj8p	Created	Created container: storage-initializer
19:47:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-blj8p	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
19:47:01	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-predictor-6cf87485dc-blj8p	AddedInterface	Add eth0 [10.134.0.23/23] from ovn-kubernetes
19:47:01 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-c0d673-predictor-6df449b465-6snbz	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-init-fail-c0d673-predictor-serving-cert" not found
19:47:02	kserve-ci-e2e-test	multus	isvc-init-fail-c0d673-predictor-6df449b465-6snbz	AddedInterface	Add eth0 [10.134.0.24/23] from ovn-kubernetes
19:47:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-blj8p	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907"
19:47:11 (x8)	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-c0d673	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:47:14 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-c0d673-predictor-6df449b465-6snbz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
19:47:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-blj8p	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
19:47:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-blj8p	Started	Started container kserve-container
19:47:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-blj8p	Created	Created container: kserve-container
19:47:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-blj8p	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" in 13.97s (13.97s including waiting). Image size: 1583067017 bytes.
19:47:19	kserve-ci-e2e-test	kubelet	isvc-init-fail-c0d673-predictor-6df449b465-6snbz	Killing	Stopping container storage-initializer
19:47:19 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-c0d673-predictor-6df449b465-6snbz	Started	Started container storage-initializer
19:47:19 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-c0d673-predictor-6df449b465-6snbz	Created	Created container: storage-initializer
19:47:20	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-predictor-5f849d69df from 0 to 1
19:47:20	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-predictor-5f849d69df	SuccessfulCreate	Created pod: isvc-predictive-sklearn-predictor-5f849d69df-6f5mk
19:47:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn": the object has been modified; please apply your changes to the latest version and try again
19:47:21	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-6f5mk	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
19:47:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-blj8p	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.318s (2.318s including waiting). Image size: 211946088 bytes.
19:47:21	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-6f5mk	Created	Created container: storage-initializer
19:47:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-blj8p	Pulling	Pulling image "quay.io/opendatahub/kserve-agent@sha256:ee489751b997752a8855e389ed75068cf003229489532801b9b57f9452d3ed82"
19:47:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-blj8p	Started	Started container kube-rbac-proxy
19:47:21	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-predictor-5f849d69df-6f5mk	AddedInterface	Add eth0 [10.134.0.25/23] from ovn-kubernetes
19:47:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-blj8p	Created	Created container: kube-rbac-proxy
19:47:21	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-6f5mk	Started	Started container storage-initializer
19:47:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-blj8p	Started	Started container agent
19:47:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-blj8p	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:ee489751b997752a8855e389ed75068cf003229489532801b9b57f9452d3ed82" in 2.909s (2.909s including waiting). Image size: 243360834 bytes.
19:47:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-blj8p	Created	Created container: agent
19:47:26	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-6f5mk	Pulling	Pulling image "kserve/predictiveserver:latest"
19:47:48	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-6f5mk	Created	Created container: kserve-container
19:47:48	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-6f5mk	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:47:48	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-6f5mk	Pulled	Successfully pulled image "kserve/predictiveserver:latest" in 22.015s (22.015s including waiting). Image size: 2439814290 bytes.
19:47:48	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-6f5mk	Started	Started container kube-rbac-proxy
19:47:48	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-6f5mk	Created	Created container: kube-rbac-proxy
19:47:48	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-6f5mk	Started	Started container kserve-container
19:48:00 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:48:00 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:48:20 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:48:20 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:48:40 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:48:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InferenceServiceReady	InferenceService [isvc-sklearn-batcher] is Ready
19:48:45 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:48:45 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:48:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-blj8p	Killing	Stopping container agent
19:48:55	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-custom-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-custom-predictor-689fcfbf74 from 0 to 1
19:48:55	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-custom-predictor-689fcfbf74	SuccessfulCreate	Created pod: isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2
19:48:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-blj8p	Killing	Stopping container kube-rbac-proxy
19:48:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher-custom": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher-custom": the object has been modified; please apply your changes to the latest version and try again
19:48:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-blj8p	Killing	Stopping container kserve-container
19:48:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2	Started	Started container storage-initializer
19:48:56	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2	AddedInterface	Add eth0 [10.134.0.26/23] from ovn-kubernetes
19:48:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
19:48:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2	Created	Created container: storage-initializer
19:49:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:ee489751b997752a8855e389ed75068cf003229489532801b9b57f9452d3ed82" already present on machine
19:49:00 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-blj8p	Unhealthy	Readiness probe failed: dial tcp 10.134.0.23:8080: connect: connection refused
19:49:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2	Created	Created container: kserve-container
19:49:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2	Started	Started container kserve-container
19:49:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:49:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2	Created	Created container: kube-rbac-proxy
19:49:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
19:49:00 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-blj8p	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
19:49:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2	Started	Started container agent
19:49:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2	Created	Created container: agent
19:49:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2	Started	Started container kube-rbac-proxy
19:49:06 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6cf87485dc-blj8p	Unhealthy	Readiness probe failed: Get "https://10.134.0.23:8643/healthz": dial tcp 10.134.0.23:8643: connect: connection refused
19:49:12 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:49:12	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	InferenceServiceReady	InferenceService [isvc-predictive-sklearn] is Ready
19:49:20 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:49:20 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:49:21	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-6f5mk	Unhealthy	Readiness probe failed: Get "https://10.134.0.25:8643/healthz": dial tcp 10.134.0.25:8643: connect: connection refused
19:49:21	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-6f5mk	Killing	Stopping container kube-rbac-proxy
19:49:21	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-6f5mk	Killing	Stopping container kserve-container
19:49:22 (x12)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-6f5mk	Unhealthy	Readiness probe failed: dial tcp 10.134.0.25:8080: connect: connection refused
19:49:25 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:49:25 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:49:26	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-whrdt	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
19:49:26	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-predictor-695f978978 from 0 to 1
19:49:26	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-predictor-695f978978	SuccessfulCreate	Created pod: isvc-predictive-xgboost-predictor-695f978978-whrdt
19:49:26	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost": the object has been modified; please apply your changes to the latest version and try again
19:49:27	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-predictor-695f978978-whrdt	AddedInterface	Add eth0 [10.134.0.27/23] from ovn-kubernetes
19:49:27	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-whrdt	Created	Created container: storage-initializer
19:49:27	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-whrdt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
19:49:27	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-whrdt	Started	Started container storage-initializer
19:49:27	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-6f5mk	Unhealthy	Readiness probe failed: Get "https://10.134.0.25:8643/healthz": context deadline exceeded
19:49:31	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-whrdt	Created	Created container: kserve-container
19:49:31	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-whrdt	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:49:31	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-whrdt	Started	Started container kserve-container
19:49:31	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-whrdt	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
19:49:32	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-whrdt	Created	Created container: kube-rbac-proxy
19:49:32	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-whrdt	Started	Started container kube-rbac-proxy
19:49:56 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:49:56 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:50:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	InferenceServiceReady	InferenceService [isvc-sklearn-batcher-custom] is Ready
19:50:18 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:50:25 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:50:25 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:50:30 (x2)	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	UpdateFailed	Failed to update status for InferenceService "message-dumper": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper": the object has been modified; please apply your changes to the latest version and try again
19:50:30	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-65ngk	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "message-dumper-predictor-serving-cert" not found
19:50:30	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper": the object has been modified; please apply your changes to the latest version and try again
19:50:30	kserve-ci-e2e-test	replicaset-controller	message-dumper-predictor-6575bcfdfd	SuccessfulCreate	Created pod: message-dumper-predictor-6575bcfdfd-65ngk
19:50:30	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2	Killing	Stopping container kube-rbac-proxy
19:50:30	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2	Killing	Stopping container agent
19:50:30	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2	Killing	Stopping container kserve-container
19:50:30	kserve-ci-e2e-test	deployment-controller	message-dumper-predictor	ScalingReplicaSet	Scaled up replica set message-dumper-predictor-6575bcfdfd from 0 to 1
19:50:31	kserve-ci-e2e-test	multus	message-dumper-predictor-6575bcfdfd-65ngk	AddedInterface	Add eth0 [10.134.0.28/23] from ovn-kubernetes
19:50:31	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-65ngk	Pulling	Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display"
19:50:33	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-65ngk	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:50:33	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-65ngk	Pulled	Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.283s (1.283s including waiting). Image size: 14813193 bytes.
19:50:33	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-65ngk	Started	Started container kube-rbac-proxy
19:50:33	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-65ngk	Created	Created container: kserve-container
19:50:33	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-65ngk	Created	Created container: kube-rbac-proxy
19:50:33	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-65ngk	Started	Started container kserve-container
19:50:38 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2	Unhealthy	Readiness probe failed: dial tcp 10.134.0.26:5000: connect: connection refused
19:50:42	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	InferenceServiceReady	InferenceService [message-dumper] is Ready
19:50:42 (x9)	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:50:48 (x4)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2	Unhealthy	Readiness probe failed: Get "https://10.134.0.26:8643/healthz": dial tcp 10.134.0.26:8643: connect: connection refused
19:50:48 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
19:50:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	InferenceServiceReady	InferenceService [isvc-predictive-xgboost] is Ready
19:50:50	kserve-ci-e2e-test	deployment-controller	isvc-logger-predictor	ScalingReplicaSet	Scaled up replica set isvc-logger-predictor-85d5cb84f6 from 0 to 1
19:50:50 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:50:50	kserve-ci-e2e-test	replicaset-controller	isvc-logger-predictor-85d5cb84f6	SuccessfulCreate	Created pod: isvc-logger-predictor-85d5cb84f6-57qgh
19:50:50	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-57qgh	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-logger-predictor-serving-cert" not found
19:50:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	UpdateFailed	Failed to update status for InferenceService "isvc-logger": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger": the object has been modified; please apply your changes to the latest version and try again
19:50:51	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-57qgh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
19:50:51	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-57qgh	Started	Started container storage-initializer
19:50:51	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-57qgh	Created	Created container: storage-initializer
19:50:51	kserve-ci-e2e-test	multus	isvc-logger-predictor-85d5cb84f6-57qgh	AddedInterface	Add eth0 [10.134.0.29/23] from ovn-kubernetes
19:50:56	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-57qgh	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:50:56	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-whrdt	Killing	Stopping container kube-rbac-proxy
19:50:56 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:50:56	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-whrdt	Killing	Stopping container kserve-container
19:50:56	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-57qgh	Created	Created container: agent
19:50:56	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-57qgh	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:ee489751b997752a8855e389ed75068cf003229489532801b9b57f9452d3ed82" already present on machine
19:50:56	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-57qgh	Started	Started container kube-rbac-proxy
19:50:56	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-57qgh	Started	Started container agent
19:50:56 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:50:56	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-57qgh	Created	Created container: kube-rbac-proxy
19:50:56	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-57qgh	Started	Started container kserve-container
19:50:56	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-57qgh	Created	Created container: kserve-container
19:50:56	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-57qgh	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
19:50:59	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-whrdt	Unhealthy	Readiness probe failed: Get "https://10.134.0.27:8643/healthz": dial tcp 10.134.0.27:8643: connect: connection refused
19:50:59 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-whrdt	Unhealthy	Readiness probe failed: dial tcp 10.134.0.27:8080: connect: connection refused
19:51:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:51:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:51:01	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-predictor-ff56bcbb5 from 0 to 1
19:51:02	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-predictor-ff56bcbb5	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6
19:51:02	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6	AddedInterface	Add eth0 [10.134.0.30/23] from ovn-kubernetes
19:51:02	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
19:51:02	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6	Created	Created container: storage-initializer
19:51:02	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6	Started	Started container storage-initializer
19:51:02 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm": the object has been modified; please apply your changes to the latest version and try again
19:51:02	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm": the object has been modified; please apply your changes to the latest version and try again
19:51:07	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6	Started	Started container kserve-container
19:51:07	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6	Created	Created container: kserve-container
19:51:07	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6	Created	Created container: kube-rbac-proxy
19:51:07	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:51:07	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
19:51:07	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6	Started	Started container kube-rbac-proxy
19:51:35 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:51:35 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:51:46 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:51:47 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:52:13	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	InferenceServiceReady	InferenceService [isvc-logger] is Ready
19:52:13 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:52:20 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:52:20 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:52:25	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-65ngk	Killing	Stopping container kserve-container
19:52:25	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v1-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v1-predictor-574b5b7495 from 0 to 1
19:52:25	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-65ngk	Killing	Stopping container kube-rbac-proxy
19:52:25	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-57qgh	Killing	Stopping container kserve-container
19:52:25	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-57qgh	Killing	Stopping container agent
19:52:25	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-57qgh	Killing	Stopping container kube-rbac-proxy
19:52:26	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v1-predictor-574b5b7495	SuccessfulCreate	Created pod: isvc-autogluon-v1-predictor-574b5b7495-df69n
19:52:26	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-df69n	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-autogluon-v1-predictor-serving-cert" not found
19:52:26	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v1": the object has been modified; please apply your changes to the latest version and try again
19:52:26	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v1": the object has been modified; please apply your changes to the latest version and try again
19:52:27	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-df69n	Started	Started container storage-initializer
19:52:27	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-df69n	Created	Created container: storage-initializer
19:52:27	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-df69n	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
19:52:27	kserve-ci-e2e-test	multus	isvc-autogluon-v1-predictor-574b5b7495-df69n	AddedInterface	Add eth0 [10.134.0.31/23] from ovn-kubernetes
19:52:32 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:52:32	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-df69n	Pulling	Pulling image "kserve/autogluonserver:latest"
19:52:32 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:52:33 (x10)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-57qgh	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
19:52:33 (x10)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-57qgh	Unhealthy	Readiness probe failed: dial tcp 10.134.0.29:8080: connect: connection refused
19:52:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm] is Ready
19:52:34 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:52:42	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6	Killing	Stopping container kserve-container
19:52:42	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6	Killing	Stopping container kube-rbac-proxy
19:52:43 (x4)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-85d5cb84f6-57qgh	Unhealthy	Readiness probe failed: Get "https://10.134.0.29:8643/healthz": dial tcp 10.134.0.29:8643: connect: connection refused
19:52:44	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6	Unhealthy	Readiness probe failed: Get "https://10.134.0.30:8643/healthz": dial tcp 10.134.0.30:8643: connect: connection refused
19:52:44 (x11)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6	Unhealthy	Readiness probe failed: dial tcp 10.134.0.30:8080: connect: connection refused
19:52:47	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-v2-predictor-745f9c8c9f from 0 to 1
19:52:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
19:52:48	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
19:52:48	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-v2-predictor-745f9c8c9f	SuccessfulCreate	Created pod: isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz
19:52:49	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz	AddedInterface	Add eth0 [10.133.0.31/23] from ovn-kubernetes
19:52:49	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63"
19:52:52	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" in 3.775s (3.775s including waiting). Image size: 306813343 bytes.
19:52:53	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz	Created	Created container: storage-initializer
19:52:53	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz	Started	Started container storage-initializer
19:52:56	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz	Pulling	Pulling image "kserve/predictiveserver:latest"
19:53:19	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz	Pulled	Successfully pulled image "kserve/predictiveserver:latest" in 22.082s (22.082s including waiting). Image size: 2439814290 bytes.
19:53:19	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz	Created	Created container: kserve-container
19:53:19	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz	Started	Started container kserve-container
19:53:19	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
19:53:21	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.657s (2.657s including waiting). Image size: 211946088 bytes.
19:53:21	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz	Started	Started container kube-rbac-proxy
19:53:21	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz	Created	Created container: kube-rbac-proxy
19:53:32 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:53:32 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:54:26 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:54:26 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:54:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-df69n	Created	Created container: kube-rbac-proxy
19:54:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-df69n	Started	Started container kube-rbac-proxy
19:54:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-df69n	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:54:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-df69n	Started	Started container kserve-container
19:54:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-df69n	Created	Created container: kserve-container
19:54:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-df69n	Pulled	Successfully pulled image "kserve/autogluonserver:latest" in 1m55.84s (1m55.84s including waiting). Image size: 8949490981 bytes.
19:54:35 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-df69n	Unhealthy	Readiness probe failed: dial tcp 10.134.0.31:8080: connect: connection refused
19:54:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	InferenceServiceReady	InferenceService [isvc-predictive-sklearn-v2] is Ready
19:54:39 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:54:41	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:54:41	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:54:45 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:54:45	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	InferenceServiceReady	InferenceService [isvc-autogluon-v1] is Ready
19:54:48 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:54:48 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:54:48	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz	Killing	Stopping container kube-rbac-proxy
19:54:48	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz	Killing	Stopping container kserve-container
19:54:49 (x6)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz	Unhealthy	Readiness probe failed: Get "http://10.133.0.31:8080/v2/models/isvc-predictive-sklearn-v2/ready": dial tcp 10.133.0.31:8080: connect: connection refused
19:54:49	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-w76bz	Unhealthy	Readiness probe failed: Get "https://10.133.0.31:8643/healthz": dial tcp 10.133.0.31:8643: connect: connection refused
19:54:52	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-df69n	Killing	Stopping container kube-rbac-proxy
19:54:52	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-df69n	Killing	Stopping container kserve-container
19:54:53	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-v2-predictor-868ff5dd6d from 0 to 1
19:54:53	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
19:54:53	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx	Created	Created container: storage-initializer
19:54:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
19:54:53	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-v2-predictor-868ff5dd6d	SuccessfulCreate	Created pod: isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx
19:54:53	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx	AddedInterface	Add eth0 [10.133.0.32/23] from ovn-kubernetes
19:54:54	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx	Started	Started container storage-initializer
19:54:57	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-predictor-9684b4cd9 from 0 to 1
19:54:57	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-predictor-9684b4cd9	SuccessfulCreate	Created pod: isvc-autogluon-v2-predictor-9684b4cd9-thssc
19:54:57	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2": the object has been modified; please apply your changes to the latest version and try again
19:54:58	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx	Created	Created container: kube-rbac-proxy
19:54:58	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:54:58	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
19:54:58	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx	Created	Created container: kserve-container
19:54:58	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx	Started	Started container kube-rbac-proxy
19:54:58	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-thssc	Created	Created container: storage-initializer
19:54:58	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-thssc	Started	Started container storage-initializer
19:54:58	kserve-ci-e2e-test	multus	isvc-autogluon-v2-predictor-9684b4cd9-thssc	AddedInterface	Add eth0 [10.134.0.32/23] from ovn-kubernetes
19:54:58	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-thssc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
19:54:58	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx	Started	Started container kserve-container
19:55:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-thssc	Created	Created container: kserve-container
19:55:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-thssc	Created	Created container: kube-rbac-proxy
19:55:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-thssc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:55:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-thssc	Started	Started container kserve-container
19:55:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-thssc	Started	Started container kube-rbac-proxy
19:55:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-thssc	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
19:55:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:55:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:55:57 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:55:57 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:56:17 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:56:17	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	InferenceServiceReady	InferenceService [isvc-predictive-xgboost-v2] is Ready
19:56:23	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx	Killing	Stopping container kserve-container
19:56:23	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx	Killing	Stopping container kube-rbac-proxy
19:56:23 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:56:23 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:56:26	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx	Unhealthy	Readiness probe failed: Get "https://10.133.0.32:8643/healthz": dial tcp 10.133.0.32:8643: connect: connection refused
19:56:27 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nrmwx	Unhealthy	Readiness probe failed: Get "http://10.133.0.32:8080/v2/models/isvc-predictive-xgboost-v2/ready": dial tcp 10.133.0.32:8080: connect: connection refused
19:56:28	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-v2-predictor-545b7cc587 from 0 to 1
19:56:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm-v2": the object has been modified; please apply your changes to the latest version and try again
19:56:28	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-v2-predictor-545b7cc587	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp
19:56:29	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp	Started	Started container storage-initializer
19:56:29	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
19:56:29	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp	AddedInterface	Add eth0 [10.133.0.33/23] from ovn-kubernetes
19:56:29	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp	Created	Created container: storage-initializer
19:56:33	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
19:56:34	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp	Created	Created container: kserve-container
19:56:34	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:56:34	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp	Started	Started container kserve-container
19:56:34	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp	Started	Started container kube-rbac-proxy
19:56:34	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp	Created	Created container: kube-rbac-proxy
19:56:43	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:56:43	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:56:52	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	InferenceServiceReady	InferenceService [isvc-autogluon-v2] is Ready
19:56:52 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:56:57 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:56:57 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:57:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-thssc	Killing	Stopping container kube-rbac-proxy
19:57:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-thssc	Killing	Stopping container kserve-container
19:57:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2-variants": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-variants": the object has been modified; please apply your changes to the latest version and try again
19:57:08	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-variants-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-variants-predictor-5c889d8844 from 0 to 1
19:57:08	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-variants-predictor-5c889d8844	SuccessfulCreate	Created pod: isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz
19:57:09	kserve-ci-e2e-test	multus	isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz	AddedInterface	Add eth0 [10.134.0.33/23] from ovn-kubernetes
19:57:09	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
19:57:09	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz	Created	Created container: storage-initializer
19:57:09	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz	Started	Started container storage-initializer
19:57:16	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
19:57:17	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz	Created	Created container: kube-rbac-proxy
19:57:17	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz	Started	Started container kube-rbac-proxy
19:57:17	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz	Created	Created container: kserve-container
19:57:17	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:57:17	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz	Started	Started container kserve-container
19:57:53 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:57:53 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:57:58 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:57:58 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:58:02	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm-v2] is Ready
19:58:02 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:58:08	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp	Killing	Stopping container kserve-container
19:58:08	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp	Killing	Stopping container kube-rbac-proxy
19:58:10	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp	Unhealthy	Readiness probe failed: Get "https://10.133.0.33:8643/healthz": dial tcp 10.133.0.33:8643: connect: connection refused
19:58:11 (x7)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-g9tzp	Unhealthy	Readiness probe failed: Get "http://10.133.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready": dial tcp 10.133.0.33:8080: connect: connection refused
19:58:13 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
19:58:13 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-scheduler": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
19:58:13	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-scheduler-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-scheduler-predictor-fd76fb6c9 from 0 to 1
19:58:13	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-scheduler-predictor-fd76fb6c9	SuccessfulCreate	Created pod: isvc-sklearn-scheduler-predictor-fd76fb6c9-png8k
19:58:14 (x6)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:58:47	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-predictor-c6b88b959	SuccessfulCreate	Created pod: isvc-sklearn-predictor-c6b88b959-rbmhx
19:58:47	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-predictor-c6b88b959 from 0 to 1
19:58:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn": the object has been modified; please apply your changes to the latest version and try again
19:58:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-rbmhx	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-predictor-serving-cert" not found
19:58:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-rbmhx	Started	Started container storage-initializer
19:58:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-rbmhx	Created	Created container: storage-initializer
19:58:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-rbmhx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
19:58:49	kserve-ci-e2e-test	multus	isvc-sklearn-predictor-c6b88b959-rbmhx	AddedInterface	Add eth0 [10.134.0.36/23] from ovn-kubernetes
19:58:51	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	InferenceServiceReady	InferenceService [isvc-autogluon-v2-variants] is Ready
19:58:51 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:58:53 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:58:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-rbmhx	Started	Started container kube-rbac-proxy
19:58:53 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:58:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-rbmhx	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
19:58:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-rbmhx	Created	Created container: kserve-container
19:58:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-rbmhx	Started	Started container kserve-container
19:58:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-rbmhx	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:58:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-rbmhx	Created	Created container: kube-rbac-proxy
19:59:02	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz	Unhealthy	Readiness probe failed: Get "http://10.134.0.33:8080/v2/models/isvc-autogluon-v2-variants/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
19:59:05	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz	Killing	Stopping container kube-rbac-proxy
19:59:05	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz	Killing	Stopping container kserve-container
19:59:10	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-noslash-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-noslash-predictor-5499c79f9f from 0 to 1
19:59:10	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-noslash-predictor-5499c79f9f	SuccessfulCreate	Created pod: isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5
19:59:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2-noslash": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-noslash": the object has been modified; please apply your changes to the latest version and try again
19:59:11	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5	Created	Created container: storage-initializer
19:59:11	kserve-ci-e2e-test	multus	isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5	AddedInterface	Add eth0 [10.133.0.34/23] from ovn-kubernetes
19:59:11	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
19:59:11	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5	Started	Started container storage-initializer
19:59:16	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5	Pulling	Pulling image "kserve/autogluonserver:latest"
19:59:32 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:59:32 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:00:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	InferenceServiceReady	InferenceService [isvc-sklearn] is Ready
20:00:10 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:00:17 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:00:17 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:00:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-rbmhx	Killing	Stopping container kserve-container
20:00:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-rbmhx	Killing	Stopping container kube-rbac-proxy
20:00:20 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-rbmhx	Unhealthy	Readiness probe failed: dial tcp 10.134.0.36:8080: connect: connection refused
20:00:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-c6b88b959-rbmhx	Unhealthy	Readiness probe failed: Get "https://10.134.0.36:8643/healthz": dial tcp 10.134.0.36:8643: connect: connection refused
20:00:23	kserve-ci-e2e-test	multus	sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6	AddedInterface	Add eth0 [10.134.0.37/23] from ovn-kubernetes
20:00:23	kserve-ci-e2e-test	deployment-controller	sklearn-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set sklearn-v2-mlserver-predictor-7f568cdc6b from 0 to 1
20:00:23	kserve-ci-e2e-test	replicaset-controller	sklearn-v2-mlserver-predictor-7f568cdc6b	SuccessfulCreate	Created pod: sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6
20:00:23	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:00:23	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "sklearn-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "sklearn-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
20:00:23	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6	Started	Started container storage-initializer
20:00:23	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6	Created	Created container: storage-initializer
20:00:27	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6	Pulling	Pulling image "docker.io/seldonio/mlserver:1.7.1"
20:01:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:01:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5	Created	Created container: kube-rbac-proxy
20:01:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5	Started	Started container kube-rbac-proxy
20:01:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5	Created	Created container: kserve-container
20:01:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5	Started	Started container kserve-container
20:01:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5	Pulled	Successfully pulled image "kserve/autogluonserver:latest" in 1m46.467s (1m46.467s including waiting). Image size: 8949490981 bytes.
20:01:25 (x9)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:01:25 (x9)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:02:10 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:02:25 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:02:38	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6	Started	Started container kserve-container
20:02:38	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6	Started	Started container kube-rbac-proxy
20:02:38	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6	Created	Created container: kube-rbac-proxy
20:02:38	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:02:38	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6	Created	Created container: kserve-container
20:02:38	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6	Pulled	Successfully pulled image "docker.io/seldonio/mlserver:1.7.1" in 2m10.351s (2m10.351s including waiting). Image size: 10890461297 bytes.
20:02:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	InferenceServiceReady	InferenceService [isvc-autogluon-v2-noslash] is Ready
20:02:42 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:02:52	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5	Unhealthy	Readiness probe failed: Get "http://10.133.0.34:8080/v2/models/isvc-autogluon-v2-noslash/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:02:57	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5	Killing	Stopping container kserve-container
20:02:57	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-vnkp5	Killing	Stopping container kube-rbac-proxy
20:03:02	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-ts-v1-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-ts-v1-predictor-6cf79d9f7d from 0 to 1
20:03:02	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-ts-v1-predictor-6cf79d9f7d	SuccessfulCreate	Created pod: isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq
20:03:02	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-ts-v1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1": the object has been modified; please apply your changes to the latest version and try again
20:03:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:03:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq	Started	Started container storage-initializer
20:03:03	kserve-ci-e2e-test	multus	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq	AddedInterface	Add eth0 [10.133.0.35/23] from ovn-kubernetes
20:03:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq	Created	Created container: storage-initializer
20:03:08	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:03:08	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:03:12	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:03:12	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq	Created	Created container: kserve-container
20:03:12	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq	Started	Started container kserve-container
20:03:12	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
20:03:12	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq	Started	Started container kube-rbac-proxy
20:03:12	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq	Created	Created container: kube-rbac-proxy
20:03:14	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
20:03:17	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:03:17	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:03:20 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq	Unhealthy	Readiness probe failed: dial tcp 10.133.0.35:8080: connect: connection refused
20:03:23 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:03:23 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:03:24	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	InferenceServiceReady	InferenceService [sklearn-v2-mlserver] is Ready
20:03:24 (x10)	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:03:30	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	InferenceServiceReady	InferenceService [isvc-autogluon-ts-v1] is Ready
20:03:30 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:03:32	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:03:32	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:03:33	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6	Killing	Stopping container kserve-container
20:03:33	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6	Killing	Stopping container kube-rbac-proxy
20:03:35	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6	Unhealthy	Readiness probe failed: Get "http://10.134.0.37:8080/v2/models/sklearn-v2-mlserver/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:03:38	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-runtime-predictor-79f4b454d7 from 0 to 1
20:03:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-runtime": the object has been modified; please apply your changes to the latest version and try again
20:03:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
20:03:38	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-runtime-predictor-79f4b454d7	SuccessfulCreate	Created pod: isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz
20:03:39 (x2)	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6	Unhealthy	Readiness probe failed: Get "https://10.134.0.37:8643/healthz": dial tcp 10.134.0.37:8643: connect: connection refused
20:03:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:03:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz	Started	Started container storage-initializer
20:03:39	kserve-ci-e2e-test	multus	isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
20:03:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz	Created	Created container: storage-initializer
20:03:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz	Created	Created container: kserve-container
20:03:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:03:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz	Created	Created container: kube-rbac-proxy
20:03:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz	Started	Started container kube-rbac-proxy
20:03:45	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq	Unhealthy	Readiness probe failed: Get "https://10.133.0.35:8643/healthz": dial tcp 10.133.0.35:8643: connect: connection refused
20:03:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz	Started	Started container kserve-container
20:03:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:03:45	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq	Killing	Stopping container kube-rbac-proxy
20:03:45	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-d7ghq	Killing	Stopping container kserve-container
20:03:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-ts-v1-noslash": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1-noslash": the object has been modified; please apply your changes to the latest version and try again
20:03:50	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585	SuccessfulCreate	Created pod: isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl
20:03:50	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-ts-v1-noslash-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-ts-v1-noslash-predictor-b54fbf585 from 0 to 1
20:03:51	kserve-ci-e2e-test	multus	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl	AddedInterface	Add eth0 [10.133.0.36/23] from ovn-kubernetes
20:03:51	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:03:51	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl	Created	Created container: storage-initializer
20:03:51	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl	Started	Started container storage-initializer
20:03:53 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz	Unhealthy	Readiness probe failed: dial tcp 10.134.0.38:8080: connect: connection refused
20:03:57	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl	Created	Created container: kube-rbac-proxy
20:03:57	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl	Started	Started container kserve-container
20:03:57	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:03:57	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl	Started	Started container kube-rbac-proxy
20:03:57	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl	Created	Created container: kserve-container
20:03:57	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
20:04:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-runtime] is Ready
20:04:03 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:04:05 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl	Unhealthy	Readiness probe failed: dial tcp 10.133.0.36:8080: connect: connection refused
20:04:05	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:04:05	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:04:08 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:04:08 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:04:10	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz	Killing	Stopping container kserve-container
20:04:10	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz	Killing	Stopping container kube-rbac-proxy
20:04:15 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:04:15	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-runtime-predictor-5f69c7d498	SuccessfulCreate	Created pod: isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5
20:04:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
20:04:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	InferenceServiceReady	InferenceService [isvc-autogluon-ts-v1-noslash] is Ready
20:04:15	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-runtime-predictor-5f69c7d498 from 0 to 1
20:04:16	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:04:16	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5	Created	Created container: storage-initializer
20:04:16	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5	Started	Started container storage-initializer
20:04:16	kserve-ci-e2e-test	multus	isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
20:04:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5	Created	Created container: kserve-container
20:04:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
20:04:20	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:04:20	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:04:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:04:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5	Started	Started container kserve-container
20:04:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5	Started	Started container kube-rbac-proxy
20:04:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5	Created	Created container: kube-rbac-proxy
20:04:26	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl	Killing	Stopping container kube-rbac-proxy
20:04:26	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-pjsjl	Killing	Stopping container kserve-container
20:04:31	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-create-v2-predictor-579784d7d7 from 0 to 1
20:04:31	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-68984b9498	SuccessfulCreate	Created pod: isvc-canary-create-predictor-68984b9498-sf9gn
20:04:31	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-v2-predictor-579784d7d7	SuccessfulCreate	Created pod: isvc-canary-create-v2-predictor-579784d7d7-fjw2s
20:04:31	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-create-predictor-68984b9498 from 0 to 2
20:04:31	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-68984b9498	SuccessfulCreate	Created pod: isvc-canary-create-predictor-68984b9498-nmsxm
20:04:32	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-fjw2s	Created	Created container: storage-initializer
20:04:32	kserve-ci-e2e-test	multus	isvc-canary-create-predictor-68984b9498-sf9gn	AddedInterface	Add eth0 [10.134.0.41/23] from ovn-kubernetes
20:04:32	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-nmsxm	Started	Started container storage-initializer
20:04:32	kserve-ci-e2e-test	multus	isvc-canary-create-predictor-68984b9498-nmsxm	AddedInterface	Add eth0 [10.133.0.37/23] from ovn-kubernetes
20:04:32	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-nmsxm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:04:32	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-nmsxm	Created	Created container: storage-initializer
20:04:32	kserve-ci-e2e-test	multus	isvc-canary-create-v2-predictor-579784d7d7-fjw2s	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
20:04:32	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-fjw2s	Started	Started container storage-initializer
20:04:32	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-sf9gn	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:04:32	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-fjw2s	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:04:32	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-sf9gn	Created	Created container: storage-initializer
20:04:32	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-sf9gn	Started	Started container storage-initializer
20:04:36	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-sf9gn	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:04:37	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-fjw2s	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:04:37	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-sf9gn	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:04:37	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-sf9gn	Started	Started container kserve-container
20:04:37	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-nmsxm	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907"
20:04:37	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-fjw2s	Started	Started container kube-rbac-proxy
20:04:37	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-fjw2s	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:04:37	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-fjw2s	Created	Created container: kserve-container
20:04:37	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-fjw2s	Started	Started container kserve-container
20:04:37	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-sf9gn	Started	Started container kube-rbac-proxy
20:04:37	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-sf9gn	Created	Created container: kube-rbac-proxy
20:04:37	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-sf9gn	Created	Created container: kserve-container
20:04:37	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-fjw2s	Created	Created container: kube-rbac-proxy
20:04:50	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-nmsxm	Created	Created container: kube-rbac-proxy
20:04:50	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-nmsxm	Started	Started container kube-rbac-proxy
20:04:50	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-nmsxm	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:04:50	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-nmsxm	Started	Started container kserve-container
20:04:50	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-nmsxm	Created	Created container: kserve-container
20:04:50	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-nmsxm	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" in 13.036s (13.036s including waiting). Image size: 1583067017 bytes.
20:04:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
20:05:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:05:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:05:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-v2-runtime] is Ready
20:05:08 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:05:15 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:05:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5	Killing	Stopping container kserve-container
20:05:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5	Killing	Stopping container kube-rbac-proxy
20:05:15 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:05:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5	Unhealthy	Readiness probe failed: Get "https://10.134.0.39:8643/healthz": dial tcp 10.134.0.39:8643: connect: connection refused
20:05:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5	Unhealthy	Readiness probe failed: Get "http://10.134.0.39:8080/v2/models/isvc-sklearn-v2-runtime/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:05:20	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-predictor-7fc6cff9ff	SuccessfulCreate	Created pod: isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt
20:05:20	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-predictor-7fc6cff9ff from 0 to 1
20:05:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
20:05:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-v2-predictor-serving-cert" not found
20:05:21	kserve-ci-e2e-test	multus	isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt	AddedInterface	Add eth0 [10.133.0.38/23] from ovn-kubernetes
20:05:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:05:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt	Created	Created container: storage-initializer
20:05:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt	Started	Started container storage-initializer
20:05:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt	Started	Started container kube-rbac-proxy
20:05:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt	Created	Created container: kube-rbac-proxy
20:05:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:05:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt	Created	Created container: kserve-container
20:05:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt	Started	Started container kserve-container
20:05:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:05:35	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:05:35	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:05:54 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	UpdateFailed	Failed to update status for InferenceService "isvc-canary-create": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again
20:05:54	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-nmsxm	Killing	Stopping container kserve-container
20:05:54	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-nmsxm	Killing	Stopping container kube-rbac-proxy
20:05:54	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-create-predictor-68984b9498 from 2 to 1
20:05:54	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-68984b9498	SuccessfulDelete	Deleted pod: isvc-canary-create-predictor-68984b9498-nmsxm
20:05:54 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again
20:05:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	InferenceServiceReady	InferenceService [isvc-canary-create] is Ready
20:05:55 (x20)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:06:01	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-fjw2s	Killing	Stopping container kserve-container
20:06:01	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-sf9gn	Killing	Stopping container kserve-container
20:06:01	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-sf9gn	Killing	Stopping container kube-rbac-proxy
20:06:01	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-fjw2s	Killing	Stopping container kube-rbac-proxy
20:06:04	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-fjw2s	Unhealthy	Readiness probe failed: Get "https://10.134.0.40:8643/healthz": dial tcp 10.134.0.40:8643: connect: connection refused
20:06:04 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-579784d7d7-fjw2s	Unhealthy	Readiness probe failed: dial tcp 10.134.0.40:8080: connect: connection refused
20:06:04 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-nmsxm	Unhealthy	Readiness probe failed: dial tcp 10.133.0.37:8080: connect: connection refused
20:06:05	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-sf9gn	Unhealthy	Readiness probe failed: Get "https://10.134.0.41:8643/healthz": dial tcp 10.134.0.41:8643: connect: connection refused
20:06:05 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-sf9gn	Unhealthy	Readiness probe failed: dial tcp 10.134.0.41:8080: connect: connection refused
20:06:06	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-cb75746f8	SuccessfulCreate	Created pod: isvc-canary-promote-predictor-cb75746f8-74sqd
20:06:06	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-predictor-cb75746f8 from 0 to 2
20:06:06	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-v2-predictor-c6f7b798f	SuccessfulCreate	Created pod: isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl
20:06:06	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-v2-predictor-c6f7b798f from 0 to 1
20:06:07	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl	Started	Started container storage-initializer
20:06:07	kserve-ci-e2e-test	multus	isvc-canary-promote-predictor-cb75746f8-74sqd	AddedInterface	Add eth0 [10.134.0.43/23] from ovn-kubernetes
20:06:07	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-5lwjh	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-promote-predictor-serving-cert" not found
20:06:07	kserve-ci-e2e-test	multus	isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl	AddedInterface	Add eth0 [10.134.0.42/23] from ovn-kubernetes
20:06:07	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-74sqd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:06:07	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-cb75746f8	SuccessfulCreate	Created pod: isvc-canary-promote-predictor-cb75746f8-5lwjh
20:06:07	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-74sqd	Created	Created container: storage-initializer
20:06:07	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl	Created	Created container: storage-initializer
20:06:07	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:06:07	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-74sqd	Started	Started container storage-initializer
20:06:08	kserve-ci-e2e-test	multus	isvc-canary-promote-predictor-cb75746f8-5lwjh	AddedInterface	Add eth0 [10.133.0.39/23] from ovn-kubernetes
20:06:08	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-5lwjh	Started	Started container storage-initializer
20:06:08	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-5lwjh	Created	Created container: storage-initializer
20:06:08	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-5lwjh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:06:11	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-74sqd	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:06:11	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-74sqd	Started	Started container kube-rbac-proxy
20:06:11	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:06:11	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl	Created	Created container: kserve-container
20:06:11	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl	Started	Started container kserve-container
20:06:11	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl	Started	Started container kube-rbac-proxy
20:06:11	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:06:11	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl	Created	Created container: kube-rbac-proxy
20:06:11	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-74sqd	Created	Created container: kube-rbac-proxy
20:06:11	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-74sqd	Created	Created container: kserve-container
20:06:11	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-74sqd	Started	Started container kserve-container
20:06:11	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-74sqd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:06:12	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-5lwjh	Created	Created container: kserve-container
20:06:12	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-5lwjh	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:06:12	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-5lwjh	Started	Started container kserve-container
20:06:12	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-5lwjh	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:06:12	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-5lwjh	Created	Created container: kube-rbac-proxy
20:06:12	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-5lwjh	Started	Started container kube-rbac-proxy
20:06:22 (x6)	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68984b9498-nmsxm	Unhealthy	Readiness probe failed: Get "https://10.133.0.37:8643/healthz": dial tcp 10.133.0.37:8643: connect: connection refused
20:06:43 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:06:43	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	InferenceServiceReady	InferenceService [isvc-sklearn-v2] is Ready
20:06:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt	Killing	Stopping container kserve-container
20:06:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt	Killing	Stopping container kube-rbac-proxy
20:06:50 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:06:50 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:06:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt	Unhealthy	Readiness probe failed: Get "https://10.133.0.38:8643/healthz": dial tcp 10.133.0.38:8643: connect: connection refused
20:06:53 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7fc6cff9ff-r7lbt	Unhealthy	Readiness probe failed: dial tcp 10.133.0.38:8080: connect: connection refused
20:06:56	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-mixed-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-mixed-predictor-6dcddf96b6 from 0 to 1
20:06:56	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-mixed-predictor-6dcddf96b6	SuccessfulCreate	Created pod: isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8
20:06:56	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-mixed": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-mixed": the object has been modified; please apply your changes to the latest version and try again
20:06:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8	Started	Started container storage-initializer
20:06:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8	Created	Created container: storage-initializer
20:06:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:06:56	kserve-ci-e2e-test	multus	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8	AddedInterface	Add eth0 [10.134.0.44/23] from ovn-kubernetes
20:07:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8	Created	Created container: kserve-container
20:07:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8	Started	Started container kube-rbac-proxy
20:07:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:07:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8	Created	Created container: kube-rbac-proxy
20:07:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8	Started	Started container kserve-container
20:07:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:07:20 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-5lwjh	Unhealthy	Readiness probe failed: dial tcp 10.133.0.39:8080: connect: connection refused
20:07:26 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:07:26 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:07:28	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-cb75746f8	SuccessfulDelete	Deleted pod: isvc-canary-promote-predictor-cb75746f8-5lwjh
20:07:28	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-promote-predictor-cb75746f8 from 2 to 1
20:07:28	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-5lwjh	Killing	Stopping container kube-rbac-proxy
20:07:28	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-5lwjh	Killing	Stopping container kserve-container
20:07:28 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again
20:07:28 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	UpdateFailed	Failed to update status for InferenceService "isvc-canary-promote": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again
20:07:30	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-5lwjh	Unhealthy	Readiness probe failed: Get "https://10.133.0.39:8643/healthz": dial tcp 10.133.0.39:8643: connect: connection refused
20:07:36	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-74sqd	Killing	Stopping container kube-rbac-proxy
20:07:36	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-74sqd	Killing	Stopping container kserve-container
20:07:37	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-sn66z	Created	Created container: storage-initializer
20:07:37	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-v2-predictor-c6f7b798f from 1 to 2
20:07:37 (x21)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:07:37	kserve-ci-e2e-test	multus	isvc-canary-promote-v2-predictor-c6f7b798f-sn66z	AddedInterface	Add eth0 [10.133.0.40/23] from ovn-kubernetes
20:07:37	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-sn66z	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:07:37	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-sn66z	Started	Started container storage-initializer
20:07:37	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-v2-predictor-c6f7b798f	SuccessfulCreate	Created pod: isvc-canary-promote-v2-predictor-c6f7b798f-sn66z
20:07:38	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-74sqd	Unhealthy	Readiness probe failed: Get "https://10.134.0.43:8643/healthz": dial tcp 10.134.0.43:8643: connect: connection refused
20:07:38 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-cb75746f8-74sqd	Unhealthy	Readiness probe failed: dial tcp 10.134.0.43:8080: connect: connection refused
20:07:42	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-sn66z	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:07:42	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-sn66z	Started	Started container kube-rbac-proxy
20:07:42	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-sn66z	Started	Started container kserve-container
20:07:42	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-sn66z	Created	Created container: kube-rbac-proxy
20:07:42	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-sn66z	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:07:42	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-sn66z	Created	Created container: kserve-container
20:08:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	InferenceServiceReady	InferenceService [isvc-sklearn-v2-mixed] is Ready
20:08:18 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:08:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8	Killing	Stopping container kserve-container
20:08:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8	Killing	Stopping container kube-rbac-proxy
20:08:26 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:08:26 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:08:28 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8	Unhealthy	Readiness probe failed: dial tcp 10.134.0.44:8080: connect: connection refused
20:08:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8	Unhealthy	Readiness probe failed: Get "https://10.134.0.44:8643/healthz": dial tcp 10.134.0.44:8643: connect: connection refused
20:08:31	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-predictor-547dfc5669	SuccessfulCreate	Created pod: isvc-tensorflow-predictor-547dfc5669-kf5wp
20:08:31	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-predictor-547dfc5669 from 0 to 1
20:08:31	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-kf5wp	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-tensorflow-predictor-serving-cert" not found
20:08:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow": the object has been modified; please apply your changes to the latest version and try again
20:08:32	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-kf5wp	Created	Created container: storage-initializer
20:08:32	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-kf5wp	Started	Started container storage-initializer
20:08:32	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-kf5wp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:08:32	kserve-ci-e2e-test	multus	isvc-tensorflow-predictor-547dfc5669-kf5wp	AddedInterface	Add eth0 [10.133.0.41/23] from ovn-kubernetes
20:08:37	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-kf5wp	Pulling	Pulling image "tensorflow/serving:2.6.2"
20:08:41	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-kf5wp	Created	Created container: kube-rbac-proxy
20:08:41	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-kf5wp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:08:41	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-kf5wp	Started	Started container kserve-container
20:08:41	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-kf5wp	Created	Created container: kserve-container
20:08:41	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-kf5wp	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 3.658s (3.658s including waiting). Image size: 425873876 bytes.
20:08:41	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-kf5wp	Started	Started container kube-rbac-proxy
20:08:46	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:08:46	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:08:48 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-kf5wp	Unhealthy	Readiness probe failed: dial tcp 10.133.0.41:8080: connect: connection refused
20:08:58	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	InferenceServiceReady	InferenceService [isvc-tensorflow] is Ready
20:08:58 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:09:00 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	InferenceServiceReady	InferenceService [isvc-canary-promote] is Ready
20:09:01	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:09:01	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:09:07	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-sn66z	Killing	Stopping container kserve-container
20:09:07	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl	Killing	Stopping container kube-rbac-proxy
20:09:07	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl	Killing	Stopping container kserve-container
20:09:07	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-sn66z	Killing	Stopping container kube-rbac-proxy
20:09:08	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl	Unhealthy	Readiness probe failed: Get "https://10.134.0.42:8643/healthz": dial tcp 10.134.0.42:8643: connect: connection refused
20:09:08 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl	Unhealthy	Readiness probe failed: dial tcp 10.134.0.42:8080: connect: connection refused
20:09:09 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-sn66z	Unhealthy	Readiness probe failed: dial tcp 10.133.0.40:8080: connect: connection refused
20:09:10	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-c6f7b798f-sn66z	Unhealthy	Readiness probe failed: Get "https://10.133.0.40:8643/healthz": dial tcp 10.133.0.40:8643: connect: connection refused
20:09:12	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-v2-predictor-6946b55967	SuccessfulCreate	Created pod: isvc-canary-rollback-v2-predictor-6946b55967-m9f2h
20:09:12	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-m9f2h	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
20:09:12	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-predictor-585bd9bcc8 from 0 to 2
20:09:12	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-585bd9bcc8	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-585bd9bcc8-88wtv
20:09:12	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-585bd9bcc8	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-585bd9bcc8-6k589
20:09:12 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	UpdateFailed	Failed to update status for InferenceService "isvc-canary-rollback": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again
20:09:12	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-v2-predictor-6946b55967 from 0 to 1
20:09:12 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again
20:09:13	kserve-ci-e2e-test	multus	isvc-canary-rollback-v2-predictor-6946b55967-m9f2h	AddedInterface	Add eth0 [10.134.0.45/23] from ovn-kubernetes
20:09:13	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-88wtv	Started	Started container storage-initializer
20:09:13	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-88wtv	Created	Created container: storage-initializer
20:09:13	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-88wtv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:09:13	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-585bd9bcc8-88wtv	AddedInterface	Add eth0 [10.133.0.42/23] from ovn-kubernetes
20:09:13	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-6k589	Started	Started container storage-initializer
20:09:13	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-6k589	Created	Created container: storage-initializer
20:09:13	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-6k589	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:09:13	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-585bd9bcc8-6k589	AddedInterface	Add eth0 [10.134.0.46/23] from ovn-kubernetes
20:09:13	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-kf5wp	Killing	Stopping container kserve-container
20:09:13	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-kf5wp	Killing	Stopping container kube-rbac-proxy
20:09:13	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-m9f2h	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:09:13	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-m9f2h	Created	Created container: storage-initializer
20:09:13	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-m9f2h	Started	Started container storage-initializer
20:09:17	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-88wtv	Created	Created container: kube-rbac-proxy
20:09:17	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-88wtv	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:09:17	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-88wtv	Started	Started container kube-rbac-proxy
20:09:17	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-88wtv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:09:17	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-88wtv	Started	Started container kserve-container
20:09:17	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-88wtv	Created	Created container: kserve-container
20:09:17	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-6k589	Started	Started container kube-rbac-proxy
20:09:17	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-6k589	Created	Created container: kube-rbac-proxy
20:09:17	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-6k589	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:09:17	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-6k589	Started	Started container kserve-container
20:09:17	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-6k589	Created	Created container: kserve-container
20:09:17	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-6k589	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:09:18	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-m9f2h	Created	Created container: kserve-container
20:09:18	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-runtime-predictor-855cfcd677 from 0 to 1
20:09:18	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-m9f2h	Started	Started container kube-rbac-proxy
20:09:18	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-m9f2h	Created	Created container: kube-rbac-proxy
20:09:18	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-m9f2h	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:09:18	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-m9f2h	Started	Started container kserve-container
20:09:18	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-m9f2h	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:09:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow-runtime": the object has been modified; please apply your changes to the latest version and try again
20:09:18	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-runtime-predictor-855cfcd677	SuccessfulCreate	Created pod: isvc-tensorflow-runtime-predictor-855cfcd677-cshjb
20:09:18	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-cshjb	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:09:18	kserve-ci-e2e-test	multus	isvc-tensorflow-runtime-predictor-855cfcd677-cshjb	AddedInterface	Add eth0 [10.134.0.47/23] from ovn-kubernetes
20:09:19	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-cshjb	Started	Started container storage-initializer
20:09:19	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-cshjb	Created	Created container: storage-initializer
20:09:24	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-cshjb	Pulling	Pulling image "tensorflow/serving:2.6.2"
20:09:27	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-cshjb	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 3.684s (3.684s including waiting). Image size: 425873876 bytes.
20:09:28	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-cshjb	Started	Started container kube-rbac-proxy
20:09:28	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-cshjb	Created	Created container: kserve-container
20:09:28	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-cshjb	Started	Started container kserve-container
20:09:28	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-cshjb	Created	Created container: kube-rbac-proxy
20:09:28	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-cshjb	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:09:35 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-cshjb	Unhealthy	Readiness probe failed: dial tcp 10.134.0.47:8080: connect: connection refused
20:09:43 (x7)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-kf5wp	Unhealthy	Readiness probe failed: Get "https://10.133.0.41:8643/healthz": dial tcp 10.133.0.41:8643: connect: connection refused
20:09:45 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:09:45	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	InferenceServiceReady	InferenceService [isvc-tensorflow-runtime] is Ready
20:09:48 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:09:48 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:10:00	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-cshjb	Killing	Stopping container kube-rbac-proxy
20:10:00	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-cshjb	Killing	Stopping container kserve-container
20:10:05	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	UpdateFailed	Failed to update status for InferenceService "isvc-triton": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-triton": the object has been modified; please apply your changes to the latest version and try again
20:10:05	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zz2pj	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-triton-predictor-serving-cert" not found
20:10:05	kserve-ci-e2e-test	replicaset-controller	isvc-triton-predictor-5d7db58676	SuccessfulCreate	Created pod: isvc-triton-predictor-5d7db58676-zz2pj
20:10:05	kserve-ci-e2e-test	deployment-controller	isvc-triton-predictor	ScalingReplicaSet	Scaled up replica set isvc-triton-predictor-5d7db58676 from 0 to 1
20:10:06	kserve-ci-e2e-test	multus	isvc-triton-predictor-5d7db58676-zz2pj	AddedInterface	Add eth0 [10.133.0.43/23] from ovn-kubernetes
20:10:06	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zz2pj	Started	Started container storage-initializer
20:10:06	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zz2pj	Created	Created container: storage-initializer
20:10:06	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zz2pj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:10:10	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zz2pj	Pulling	Pulling image "nvcr.io/nvidia/tritonserver:23.05-py3"
20:10:30 (x7)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-cshjb	Unhealthy	Readiness probe failed: Get "https://10.134.0.47:8643/healthz": dial tcp 10.134.0.47:8643: connect: connection refused
20:10:35 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-88wtv	Unhealthy	Readiness probe failed: dial tcp 10.133.0.42:8080: connect: connection refused
20:10:45 (x21)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:10:45	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-585bd9bcc8	SuccessfulDelete	Deleted pod: isvc-canary-rollback-predictor-585bd9bcc8-88wtv
20:10:45	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-rollback-predictor-585bd9bcc8 from 2 to 1
20:10:46	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-88wtv	Killing	Stopping container kserve-container
20:10:46	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-88wtv	Killing	Stopping container kube-rbac-proxy
20:10:50	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-88wtv	Unhealthy	Readiness probe failed: Get "https://10.133.0.42:8643/healthz": dial tcp 10.133.0.42:8643: connect: connection refused
20:10:52	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-585bd9bcc8	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-585bd9bcc8-l6xcg
20:10:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-m9f2h	Killing	Stopping container kserve-container
20:10:52	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-predictor-585bd9bcc8 from 1 to 2
20:10:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-m9f2h	Killing	Stopping container kube-rbac-proxy
20:10:53	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-l6xcg	Started	Started container storage-initializer
20:10:53	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-585bd9bcc8-l6xcg	AddedInterface	Add eth0 [10.133.0.44/23] from ovn-kubernetes
20:10:53	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-l6xcg	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:10:53	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-l6xcg	Created	Created container: storage-initializer
20:10:55 (x12)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-m9f2h	Unhealthy	Readiness probe failed: dial tcp 10.134.0.45:8080: connect: connection refused
20:10:55	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6946b55967-m9f2h	Unhealthy	Readiness probe failed: Get "https://10.134.0.45:8643/healthz": dial tcp 10.134.0.45:8643: connect: connection refused
20:11:25	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-l6xcg	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:11:35	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-l6xcg	Created	Created container: kserve-container
20:11:35	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-l6xcg	Started	Started container kserve-container
20:11:35	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-l6xcg	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:12:04	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-l6xcg	Started	Started container kube-rbac-proxy
20:12:04	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-l6xcg	Created	Created container: kube-rbac-proxy
20:12:05	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zz2pj	Pulled	Successfully pulled image "nvcr.io/nvidia/tritonserver:23.05-py3" in 1m54.851s (1m54.851s including waiting). Image size: 12907074623 bytes.
20:12:05 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:12:05	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zz2pj	Created	Created container: kube-rbac-proxy
20:12:05 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:12:05	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zz2pj	Created	Created container: kserve-container
20:12:05	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zz2pj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:12:05	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zz2pj	Started	Started container kserve-container
20:12:06	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zz2pj	Started	Started container kube-rbac-proxy
20:12:08 (x2)	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zz2pj	Unhealthy	Readiness probe failed: dial tcp 10.133.0.43:8080: connect: connection refused
20:12:13 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:12:13	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	InferenceServiceReady	InferenceService [isvc-triton] is Ready
20:12:17	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zz2pj	Killing	Stopping container kube-rbac-proxy
20:12:17	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zz2pj	Killing	Stopping container kserve-container
20:12:18	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-zz2pj	Unhealthy	Readiness probe failed: Get "https://10.133.0.43:8643/healthz": dial tcp 10.133.0.43:8643: connect: connection refused
20:12:22	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-predictor-7566d65bf7 from 0 to 1
20:12:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost": the object has been modified; please apply your changes to the latest version and try again
20:12:22	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-predictor-7566d65bf7	SuccessfulCreate	Created pod: isvc-xgboost-predictor-7566d65bf7-ptfg8
20:12:23	kserve-ci-e2e-test	multus	isvc-xgboost-predictor-7566d65bf7-ptfg8	AddedInterface	Add eth0 [10.134.0.48/23] from ovn-kubernetes
20:12:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ptfg8	Started	Started container storage-initializer
20:12:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ptfg8	Created	Created container: storage-initializer
20:12:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ptfg8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:12:28	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ptfg8	Pulling	Pulling image "kserve/xgbserver:latest"
20:12:45	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ptfg8	Started	Started container kube-rbac-proxy
20:12:45	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ptfg8	Created	Created container: kserve-container
20:12:45	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ptfg8	Created	Created container: kube-rbac-proxy
20:12:45	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ptfg8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:12:45	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ptfg8	Started	Started container kserve-container
20:12:45	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ptfg8	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 17.652s (17.652s including waiting). Image size: 1436472579 bytes.
20:12:52 (x7)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-l6xcg	Unhealthy	Readiness probe failed: dial tcp 10.133.0.44:8080: connect: connection refused
20:13:02 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	InferenceServiceReady	InferenceService [isvc-canary-rollback] is Ready
20:13:12	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-6k589	Killing	Stopping container kserve-container
20:13:12	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-l6xcg	Killing	Stopping container kube-rbac-proxy
20:13:12	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-l6xcg	Killing	Stopping container kserve-container
20:13:12	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-6k589	Killing	Stopping container kube-rbac-proxy
20:13:15 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-6k589	Unhealthy	Readiness probe failed: dial tcp 10.134.0.46:8080: connect: connection refused
20:13:15	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-585bd9bcc8-6k589	Unhealthy	Readiness probe failed: Get "https://10.134.0.46:8643/healthz": dial tcp 10.134.0.46:8643: connect: connection refused
20:13:17	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-stop-predictor-5d5b899946 from 0 to 2
20:13:17	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-v2-predictor-5f87c564d7	SuccessfulCreate	Created pod: isvc-canary-stop-v2-predictor-5f87c564d7-64b29
20:13:17	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-stop-v2-predictor-5f87c564d7 from 0 to 1
20:13:17	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-5d5b899946	SuccessfulCreate	Created pod: isvc-canary-stop-predictor-5d5b899946-lvdqd
20:13:17	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-5d5b899946	SuccessfulCreate	Created pod: isvc-canary-stop-predictor-5d5b899946-9xkt2
20:13:18	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-lvdqd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:13:18	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-lvdqd	Created	Created container: storage-initializer
20:13:18	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-64b29	Started	Started container storage-initializer
20:13:18	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-64b29	Created	Created container: storage-initializer
20:13:18	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-64b29	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:13:18	kserve-ci-e2e-test	multus	isvc-canary-stop-v2-predictor-5f87c564d7-64b29	AddedInterface	Add eth0 [10.133.0.45/23] from ovn-kubernetes
20:13:18	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-lvdqd	Started	Started container storage-initializer
20:13:18	kserve-ci-e2e-test	multus	isvc-canary-stop-predictor-5d5b899946-9xkt2	AddedInterface	Add eth0 [10.133.0.46/23] from ovn-kubernetes
20:13:18	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-9xkt2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:13:18	kserve-ci-e2e-test	multus	isvc-canary-stop-predictor-5d5b899946-lvdqd	AddedInterface	Add eth0 [10.134.0.49/23] from ovn-kubernetes
20:13:18	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-9xkt2	Created	Created container: storage-initializer
20:13:18	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-9xkt2	Started	Started container storage-initializer
20:13:22	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-9xkt2	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:13:22 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:13:22	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-64b29	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:13:22 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:13:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-64b29	Created	Created container: kserve-container
20:13:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-64b29	Started	Started container kube-rbac-proxy
20:13:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-9xkt2	Created	Created container: kserve-container
20:13:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-9xkt2	Started	Started container kserve-container
20:13:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-lvdqd	Created	Created container: kube-rbac-proxy
20:13:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-64b29	Started	Started container kserve-container
20:13:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-64b29	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:13:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-64b29	Created	Created container: kube-rbac-proxy
20:13:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-lvdqd	Started	Started container kube-rbac-proxy
20:13:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-lvdqd	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:13:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-9xkt2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:13:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-9xkt2	Created	Created container: kube-rbac-proxy
20:13:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-9xkt2	Started	Started container kube-rbac-proxy
20:13:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-lvdqd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:13:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-lvdqd	Started	Started container kserve-container
20:13:23	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-lvdqd	Created	Created container: kserve-container
20:13:53 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ptfg8	Unhealthy	Readiness probe failed: dial tcp 10.134.0.48:8080: connect: connection refused
20:14:03 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:14:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	InferenceServiceReady	InferenceService [isvc-xgboost] is Ready
20:14:07 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:14:07 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:14:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ptfg8	Killing	Stopping container kserve-container
20:14:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ptfg8	Killing	Stopping container kube-rbac-proxy
20:14:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
20:14:18	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-mlserver-predictor-7b55b84f54 from 0 to 1
20:14:18	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-mlserver-predictor-7b55b84f54	SuccessfulCreate	Created pod: isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp
20:14:19	kserve-ci-e2e-test	multus	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp	AddedInterface	Add eth0 [10.134.0.50/23] from ovn-kubernetes
20:14:19	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:14:19	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp	Created	Created container: storage-initializer
20:14:19	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp	Started	Started container storage-initializer
20:14:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp	Started	Started container kube-rbac-proxy
20:14:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp	Created	Created container: kserve-container
20:14:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
20:14:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:14:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp	Created	Created container: kube-rbac-proxy
20:14:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp	Started	Started container kserve-container
20:14:33 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-lvdqd	Unhealthy	Readiness probe failed: dial tcp 10.134.0.49:8080: connect: connection refused
20:14:36 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-9xkt2	Unhealthy	Readiness probe failed: dial tcp 10.133.0.46:8080: connect: connection refused
20:14:42 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	UpdateFailed	Failed to update status for InferenceService "isvc-canary-stop": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again
20:14:42	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-9xkt2	Killing	Stopping container kube-rbac-proxy
20:14:42	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-9xkt2	Killing	Stopping container kserve-container
20:14:42 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again
20:14:42	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-stop-predictor-5d5b899946 from 2 to 1
20:14:42	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-5d5b899946	SuccessfulDelete	Deleted pod: isvc-canary-stop-predictor-5d5b899946-9xkt2
20:14:43	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	InferenceServiceReady	InferenceService [isvc-canary-stop] is Ready
20:14:43 (x21)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:14:44	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-9xkt2	Unhealthy	Readiness probe failed: Get "https://10.133.0.46:8643/healthz": dial tcp 10.133.0.46:8643: connect: connection refused
20:14:47	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-lvdqd	Killing	Stopping container kube-rbac-proxy
20:14:47	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-lvdqd	Killing	Stopping container kserve-container
20:14:47	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-64b29	Killing	Stopping container kserve-container
20:14:47	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-64b29	Killing	Stopping container kube-rbac-proxy
20:14:48	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:14:48	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:14:49	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-64b29	Unhealthy	Readiness probe failed: Get "https://10.133.0.45:8643/healthz": dial tcp 10.133.0.45:8643: connect: connection refused
20:14:51	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5d5b899946-lvdqd	Unhealthy	Readiness probe failed: Get "https://10.134.0.49:8643/healthz": dial tcp 10.134.0.49:8643: connect: connection refused
20:14:51 (x11)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5f87c564d7-64b29	Unhealthy	Readiness probe failed: dial tcp 10.133.0.45:8080: connect: connection refused
20:14:57	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-predictor-54f85ddf88 from 0 to 1
20:14:58	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nzvr5	Started	Started container storage-initializer
20:14:58	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm": the object has been modified; please apply your changes to the latest version and try again
20:14:58	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-predictor-54f85ddf88	SuccessfulCreate	Created pod: isvc-lightgbm-predictor-54f85ddf88-nzvr5
20:14:58	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nzvr5	Created	Created container: storage-initializer
20:14:58	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nzvr5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:14:58	kserve-ci-e2e-test	multus	isvc-lightgbm-predictor-54f85ddf88-nzvr5	AddedInterface	Add eth0 [10.133.0.47/23] from ovn-kubernetes
20:15:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	InferenceServiceReady	InferenceService [isvc-xgboost-v2-mlserver] is Ready
20:15:00 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:15:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:15:03	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nzvr5	Pulling	Pulling image "kserve/lgbserver:latest"
20:15:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:15:08	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp	Killing	Stopping container kserve-container
20:15:08	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp	Killing	Stopping container kube-rbac-proxy
20:15:09	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nzvr5	Pulled	Successfully pulled image "kserve/lgbserver:latest" in 6.434s (6.434s including waiting). Image size: 644656917 bytes.
20:15:09	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nzvr5	Started	Started container kserve-container
20:15:09	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nzvr5	Created	Created container: kserve-container
20:15:09	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nzvr5	Created	Created container: kube-rbac-proxy
20:15:09	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nzvr5	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:15:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nzvr5	Started	Started container kube-rbac-proxy
20:15:13	kserve-ci-e2e-test	deployment-controller	xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set xgboost-v2-mlserver-predictor-59944bcf4c from 0 to 1
20:15:13	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
20:15:13	kserve-ci-e2e-test	replicaset-controller	xgboost-v2-mlserver-predictor-59944bcf4c	SuccessfulCreate	Created pod: xgboost-v2-mlserver-predictor-59944bcf4c-xtj59
20:15:14	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-xtj59	Started	Started container storage-initializer
20:15:14	kserve-ci-e2e-test	multus	xgboost-v2-mlserver-predictor-59944bcf4c-xtj59	AddedInterface	Add eth0 [10.134.0.51/23] from ovn-kubernetes
20:15:14	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-xtj59	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:15:14	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-xtj59	Created	Created container: storage-initializer
20:15:15 (x2)	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp	Unhealthy	Readiness probe failed: Get "https://10.134.0.50:8643/healthz": dial tcp 10.134.0.50:8643: connect: connection refused
20:15:18	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-xtj59	Started	Started container kserve-container
20:15:18	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-xtj59	Started	Started container kube-rbac-proxy
20:15:18	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-xtj59	Created	Created container: kserve-container
20:15:18	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-xtj59	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
20:15:18	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-xtj59	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:15:18	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-xtj59	Created	Created container: kube-rbac-proxy
20:15:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:15:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:15:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:15:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:15:55	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	InferenceServiceReady	InferenceService [xgboost-v2-mlserver] is Ready
20:15:55 (x11)	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:15:58	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:15:58	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:16:03	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-xtj59	Killing	Stopping container kube-rbac-proxy
20:16:03	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-xtj59	Killing	Stopping container kserve-container
20:16:05	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-xtj59	Unhealthy	Readiness probe failed: Get "https://10.134.0.51:8643/healthz": dial tcp 10.134.0.51:8643: connect: connection refused
20:16:09	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-h8hwc	Started	Started container storage-initializer
20:16:09	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-runtime-predictor-54dd488c85	SuccessfulCreate	Created pod: isvc-xgboost-runtime-predictor-54dd488c85-h8hwc
20:16:09	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-runtime-predictor-54dd488c85 from 0 to 1
20:16:09	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-runtime": the object has been modified; please apply your changes to the latest version and try again
20:16:09	kserve-ci-e2e-test	multus	isvc-xgboost-runtime-predictor-54dd488c85-h8hwc	AddedInterface	Add eth0 [10.134.0.52/23] from ovn-kubernetes
20:16:09	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-h8hwc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:16:09	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-h8hwc	Created	Created container: storage-initializer
20:16:14	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-h8hwc	Created	Created container: kube-rbac-proxy
20:16:14	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-h8hwc	Pulled	Container image "kserve/xgbserver:latest" already present on machine
20:16:14	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-h8hwc	Created	Created container: kserve-container
20:16:14	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-h8hwc	Started	Started container kserve-container
20:16:14	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-h8hwc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:16:14	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-h8hwc	Started	Started container kube-rbac-proxy
20:16:52 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:16:52	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	InferenceServiceReady	InferenceService [isvc-lightgbm] is Ready
20:16:54 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:16:54 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:16:58 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:16:58 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:16:58	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nzvr5	Killing	Stopping container kube-rbac-proxy
20:16:58	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nzvr5	Killing	Stopping container kserve-container
20:17:01 (x13)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nzvr5	Unhealthy	Readiness probe failed: dial tcp 10.133.0.47:8080: connect: connection refused
20:17:02	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nzvr5	Unhealthy	Readiness probe failed: Get "https://10.133.0.47:8643/healthz": dial tcp 10.133.0.47:8643: connect: connection refused
20:17:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-runtime": the object has been modified; please apply your changes to the latest version and try again
20:17:03	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-runtime-predictor-577f7db7cb from 0 to 1
20:17:03	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-runtime-predictor-577f7db7cb	SuccessfulCreate	Created pod: isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55
20:17:04	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55	Started	Started container storage-initializer
20:17:04	kserve-ci-e2e-test	multus	isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55	AddedInterface	Add eth0 [10.134.0.53/23] from ovn-kubernetes
20:17:04	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:17:04	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55	Created	Created container: storage-initializer
20:17:08	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55	Pulling	Pulling image "kserve/lgbserver:latest"
20:17:14	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55	Pulled	Successfully pulled image "kserve/lgbserver:latest" in 6.662s (6.662s including waiting). Image size: 644656917 bytes.
20:17:15	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:17:15	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55	Started	Started container kserve-container
20:17:15	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55	Created	Created container: kserve-container
20:17:15	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55	Created	Created container: kube-rbac-proxy
20:17:15	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55	Started	Started container kube-rbac-proxy
20:17:31 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:17:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-runtime] is Ready
20:17:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:17:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:17:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-h8hwc	Killing	Stopping container kube-rbac-proxy
20:17:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-h8hwc	Killing	Stopping container kserve-container
20:17:41 (x10)	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-h8hwc	Unhealthy	Readiness probe failed: dial tcp 10.134.0.52:8080: connect: connection refused
20:17:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-h8hwc	Unhealthy	Readiness probe failed: Get "https://10.134.0.52:8643/healthz": dial tcp 10.134.0.52:8643: connect: connection refused
20:17:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
20:17:44	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-runtime-predictor-7787c55dc9 from 0 to 1
20:17:45	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
20:17:45	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-runtime-predictor-7787c55dc9	SuccessfulCreate	Created pod: isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8
20:17:46	kserve-ci-e2e-test	multus	isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8	AddedInterface	Add eth0 [10.134.0.54/23] from ovn-kubernetes
20:17:46	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:17:46	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8	Created	Created container: storage-initializer
20:17:46	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8	Started	Started container storage-initializer
20:17:48 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:17:48 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:17:50	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:17:50	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
20:17:50	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8	Created	Created container: kserve-container
20:17:50	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8	Started	Started container kserve-container
20:17:50	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8	Created	Created container: kube-rbac-proxy
20:17:50	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8	Started	Started container kube-rbac-proxy
20:17:59	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:17:59	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:18:28	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
20:18:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-v2-runtime] is Ready
20:18:38 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:18:45	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8	Killing	Stopping container kserve-container
20:18:45	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8	Killing	Stopping container kube-rbac-proxy
20:18:45 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:18:45 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:18:46	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-runtime] is Ready
20:18:46 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:18:48 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:18:48 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:18:48	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8	Unhealthy	Readiness probe failed: Get "http://10.134.0.54:8080/v2/models/isvc-xgboost-v2-runtime/ready": dial tcp 10.134.0.54:8080: connect: connection refused
20:18:48	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8	Unhealthy	Readiness probe failed: Get "https://10.134.0.54:8643/healthz": dial tcp 10.134.0.54:8643: connect: connection refused
20:18:50	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-hm8pd	Started	Started container storage-initializer
20:18:50	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-predictor-fcc668bc from 0 to 1
20:18:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
20:18:50	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-hm8pd	Created	Created container: storage-initializer
20:18:50	kserve-ci-e2e-test	multus	isvc-xgboost-v2-predictor-fcc668bc-hm8pd	AddedInterface	Add eth0 [10.133.0.48/23] from ovn-kubernetes
20:18:50	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-hm8pd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:18:50	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-predictor-fcc668bc	SuccessfulCreate	Created pod: isvc-xgboost-v2-predictor-fcc668bc-hm8pd
20:18:55	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-hm8pd	Pulling	Pulling image "kserve/xgbserver:latest"
20:18:55	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55	Killing	Stopping container kube-rbac-proxy
20:18:55	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55	Killing	Stopping container kserve-container
20:18:56 (x12)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55	Unhealthy	Readiness probe failed: dial tcp 10.134.0.53:8080: connect: connection refused
20:18:57	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55	Unhealthy	Readiness probe failed: Get "https://10.134.0.53:8643/healthz": dial tcp 10.134.0.53:8643: connect: connection refused
20:19:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:19:00	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5	SuccessfulCreate	Created pod: isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t
20:19:00	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t	AddedInterface	Add eth0 [10.134.0.55/23] from ovn-kubernetes
20:19:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t	Created	Created container: storage-initializer
20:19:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
20:19:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t	Started	Started container storage-initializer
20:19:00	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-runtime-predictor-7f4957c5d5 from 0 to 1
20:19:04	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
20:19:04	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t	Created	Created container: kserve-container
20:19:04	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t	Started	Started container kserve-container
20:19:04	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:19:04	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t	Created	Created container: kube-rbac-proxy
20:19:04	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t	Started	Started container kube-rbac-proxy
20:19:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-hm8pd	Started	Started container kserve-container
20:19:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-hm8pd	Started	Started container kube-rbac-proxy
20:19:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-hm8pd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:19:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-hm8pd	Created	Created container: kserve-container
20:19:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-hm8pd	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 18.38s (18.38s including waiting). Image size: 1436472579 bytes.
20:19:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-hm8pd	Created	Created container: kube-rbac-proxy
20:19:15	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:19:15	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:19:42 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:19:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-runtime] is Ready
20:19:45 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:19:45 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:19:50	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t	Killing	Stopping container kserve-container
20:19:50	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t	Killing	Stopping container kube-rbac-proxy
20:19:50 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:19:50 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:19:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
20:19:55	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5 from 0 to 1
20:19:55	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5	SuccessfulCreate	Created pod: isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4
20:19:56	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4	Started	Started container storage-initializer
20:19:56	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4	Created	Created container: storage-initializer
20:19:56	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4	AddedInterface	Add eth0 [10.134.0.56/23] from ovn-kubernetes
20:19:56	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:20:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4	Created	Created container: kserve-container
20:20:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4	Pulled	Container image "kserve/lgbserver:latest" already present on machine
20:20:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4	Started	Started container kserve-container
20:20:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:20:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4	Created	Created container: kube-rbac-proxy
20:20:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4	Started	Started container kube-rbac-proxy
20:20:08 (x3)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4	Unhealthy	Readiness probe failed: dial tcp 10.134.0.56:8080: connect: connection refused
20:20:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-kserve] is Ready
20:20:18 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:20:25	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4	Killing	Stopping container kube-rbac-proxy
20:20:25	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4	Killing	Stopping container kserve-container
20:20:25 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:20:25 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:20:30	kserve-ci-e2e-test	deployment-controller	isvc-mlflow-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-mlflow-v2-runtime-predictor-6456f94494 from 0 to 1
20:20:30	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-mlflow-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-mlflow-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
20:20:30	kserve-ci-e2e-test	replicaset-controller	isvc-mlflow-v2-runtime-predictor-6456f94494	SuccessfulCreate	Created pod: isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf
20:20:31	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf	Started	Started container storage-initializer
20:20:31	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf	Created	Created container: storage-initializer
20:20:31	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:20:31	kserve-ci-e2e-test	multus	isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf	AddedInterface	Add eth0 [10.134.0.57/23] from ovn-kubernetes
20:20:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	InferenceServiceReady	InferenceService [isvc-xgboost-v2] is Ready
20:20:31 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:20:35 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:20:35 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:20:36	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf	Created	Created container: kube-rbac-proxy
20:20:36	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf	Created	Created container: kserve-container
20:20:36	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf	Started	Started container kserve-container
20:20:36	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:20:36	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
20:20:36	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf	Started	Started container kube-rbac-proxy
20:20:40	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-hm8pd	Killing	Stopping container kserve-container
20:20:40	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-hm8pd	Killing	Stopping container kube-rbac-proxy
20:20:41 (x10)	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-hm8pd	Unhealthy	Readiness probe failed: dial tcp 10.133.0.48:8080: connect: connection refused
20:20:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-hm8pd	Unhealthy	Readiness probe failed: Get "https://10.133.0.48:8643/healthz": dial tcp 10.133.0.48:8643: connect: connection refused
20:20:45	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:20:45	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:20:46	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3": the object has been modified; please apply your changes to the latest version and try again
20:20:46	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-predictor-6c7db45b94	SuccessfulCreate	Created pod: isvc-sklearn-s3-predictor-6c7db45b94-xfs6b
20:20:46	kserve-ci-e2e-test	multus	isvc-sklearn-s3-predictor-6c7db45b94-xfs6b	AddedInterface	Add eth0 [10.134.0.58/23] from ovn-kubernetes
20:20:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-xfs6b	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:20:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-xfs6b	Started	Started container storage-initializer
20:20:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-xfs6b	Created	Created container: storage-initializer
20:20:46	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-predictor-6c7db45b94 from 0 to 1
20:20:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-xfs6b	Started	Started container kube-rbac-proxy
20:20:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-xfs6b	Created	Created container: kube-rbac-proxy
20:20:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-xfs6b	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:20:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-xfs6b	Created	Created container: kserve-container
20:20:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-xfs6b	Started	Started container kserve-container
20:20:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-xfs6b	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:21:13 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:21:13	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	InferenceServiceReady	InferenceService [isvc-mlflow-v2-runtime] is Ready
20:21:15 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:21:15 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:21:16 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:21:16 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:21:20	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf	Killing	Stopping container kube-rbac-proxy
20:21:20	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf	Killing	Stopping container kserve-container
20:21:25	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-mcp-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-mcp-predictor-78bfb5fcd9 from 0 to 1
20:21:26	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-mcp": the object has been modified; please apply your changes to the latest version and try again
20:21:26	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-mcp-predictor-78bfb5fcd9	SuccessfulCreate	Created pod: isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn
20:21:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
20:21:26 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-mcp": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-mcp": the object has been modified; please apply your changes to the latest version and try again
20:21:27	kserve-ci-e2e-test	multus	isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn	AddedInterface	Add eth0 [10.134.0.59/23] from ovn-kubernetes
20:21:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:21:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn	Started	Started container storage-initializer
20:21:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn	Created	Created container: storage-initializer
20:21:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn	Started	Started container kserve-container
20:21:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:21:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn	Created	Created container: kserve-container
20:21:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn	Pulling	Pulling image "quay.io/opendatahub/kserve-agent:latest"
20:21:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent:latest" in 2.76s (2.76s including waiting). Image size: 239481897 bytes.
20:21:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn	Created	Created container: kserve-agent
20:21:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn	Started	Started container kserve-agent
20:21:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:21:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn	Created	Created container: kube-rbac-proxy
20:21:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn	Started	Started container kube-rbac-proxy
20:21:56 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-xfs6b	Unhealthy	Readiness probe failed: dial tcp 10.134.0.58:8080: connect: connection refused
20:22:06 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:22:06	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	InferenceServiceReady	InferenceService [isvc-sklearn-s3] is Ready
20:22:16	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl
20:22:16 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:22:16	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-xfs6b	Killing	Stopping container kserve-container
20:22:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-pass": the object has been modified; please apply your changes to the latest version and try again
20:22:16	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-pass-predictor-649878f5d from 0 to 1
20:22:16	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c7db45b94-xfs6b	Killing	Stopping container kube-rbac-proxy
20:22:16 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:22:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:22:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl	Created	Created container: storage-initializer
20:22:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl	Started	Started container storage-initializer
20:22:17	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl	AddedInterface	Add eth0 [10.134.0.60/23] from ovn-kubernetes
20:22:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl	Created	Created container: kube-rbac-proxy
20:22:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:22:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl	Created	Created container: kserve-container
20:22:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl	Started	Started container kube-rbac-proxy
20:22:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl	Started	Started container kserve-container
20:22:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:22:41 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:22:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	InferenceServiceReady	InferenceService [isvc-sklearn-mcp] is Ready
20:22:46 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:22:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn	Killing	Stopping container kserve-agent
20:22:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn	Killing	Stopping container kube-rbac-proxy
20:22:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn	Killing	Stopping container kserve-container
20:22:46 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:22:51	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-5c6v8	Started	Started container storage-initializer
20:22:51	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	UpdateFailed	Failed to update status for InferenceService "isvc-paddle": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle": the object has been modified; please apply your changes to the latest version and try again
20:22:51	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-5c6v8	Created	Created container: storage-initializer
20:22:51	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-5c6v8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:22:51	kserve-ci-e2e-test	multus	isvc-paddle-predictor-6d5c9fd4f6-5c6v8	AddedInterface	Add eth0 [10.134.0.61/23] from ovn-kubernetes
20:22:51	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-predictor-6d5c9fd4f6	SuccessfulCreate	Created pod: isvc-paddle-predictor-6d5c9fd4f6-5c6v8
20:22:51	kserve-ci-e2e-test	deployment-controller	isvc-paddle-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-predictor-6d5c9fd4f6 from 0 to 1
20:22:56	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-5c6v8	Pulling	Pulling image "kserve/paddleserver:latest"
20:23:07	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-5c6v8	Pulled	Successfully pulled image "kserve/paddleserver:latest" in 11.091s (11.091s including waiting). Image size: 1204389121 bytes.
20:23:07	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-5c6v8	Created	Created container: kserve-container
20:23:07	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-5c6v8	Started	Started container kserve-container
20:23:07	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-5c6v8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:23:08	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-5c6v8	Created	Created container: kube-rbac-proxy
20:23:08	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-5c6v8	Started	Started container kube-rbac-proxy
20:23:11 (x4)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn	Unhealthy	Readiness probe failed: Get "http://10.134.0.59:8080/v1/models/isvc-sklearn-mcp": dial tcp 10.134.0.59:8080: connect: connection refused
20:23:11 (x6)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn	Unhealthy	Readiness probe failed: Get "https://10.134.0.59:8643/healthz": dial tcp 10.134.0.59:8643: connect: connection refused
20:23:21 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:23:21 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:23:25 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl	Unhealthy	Readiness probe failed: dial tcp 10.134.0.60:8080: connect: connection refused
20:23:31 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:23:31 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:23:34 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:23:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-global-pass] is Ready
20:23:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl	Killing	Stopping container kserve-container
20:23:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl	Killing	Stopping container kube-rbac-proxy
20:23:37	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-fail": the object has been modified; please apply your changes to the latest version and try again
20:23:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
20:23:37	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg
20:23:37	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6 from 0 to 1
20:23:38	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg	AddedInterface	Add eth0 [10.134.0.62/23] from ovn-kubernetes
20:23:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl	Unhealthy	Readiness probe failed: Get "https://10.134.0.60:8643/healthz": dial tcp 10.134.0.60:8643: connect: connection refused
20:23:46 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:23:46 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:23:46 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg	Started	Started container storage-initializer
20:23:46 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg	Created	Created container: storage-initializer
20:23:48	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb
20:23:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg	Killing	Stopping container storage-initializer
20:23:48	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb from 0 to 1
20:23:48	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-pass": the object has been modified; please apply your changes to the latest version and try again
20:23:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb	Started	Started container storage-initializer
20:23:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb	Created	Created container: storage-initializer
20:23:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:23:49	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb	AddedInterface	Add eth0 [10.134.0.63/23] from ovn-kubernetes
20:23:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:23:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:23:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb	Created	Created container: kserve-container
20:23:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb	Started	Started container kserve-container
20:23:51 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:23:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb	Created	Created container: kube-rbac-proxy
20:23:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb	Started	Started container kube-rbac-proxy
20:23:51 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:23:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	InferenceServiceReady	InferenceService [isvc-paddle] is Ready
20:23:55 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:24:02	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-5c6v8	Killing	Stopping container kserve-container
20:24:02	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-5c6v8	Killing	Stopping container kube-rbac-proxy
20:24:04 (x7)	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-5c6v8	Unhealthy	Readiness probe failed: dial tcp 10.134.0.61:8080: connect: connection refused
20:24:04	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-5c6v8	Unhealthy	Readiness probe failed: Get "https://10.134.0.61:8643/healthz": dial tcp 10.134.0.61:8643: connect: connection refused
20:24:07	kserve-ci-e2e-test	deployment-controller	isvc-paddle-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-runtime-predictor-58b99fff76 from 0 to 1
20:24:07	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-runtime": the object has been modified; please apply your changes to the latest version and try again
20:24:07	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-runtime-predictor-58b99fff76	SuccessfulCreate	Created pod: isvc-paddle-runtime-predictor-58b99fff76-2xpx4
20:24:08	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-2xpx4	Started	Started container storage-initializer
20:24:08	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-2xpx4	Created	Created container: storage-initializer
20:24:08	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-2xpx4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:24:08	kserve-ci-e2e-test	multus	isvc-paddle-runtime-predictor-58b99fff76-2xpx4	AddedInterface	Add eth0 [10.134.0.64/23] from ovn-kubernetes
20:24:13	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-2xpx4	Pulled	Container image "kserve/paddleserver:latest" already present on machine
20:24:13	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-2xpx4	Created	Created container: kserve-container
20:24:13	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-2xpx4	Started	Started container kserve-container
20:24:13	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-2xpx4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:24:13	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-2xpx4	Created	Created container: kube-rbac-proxy
20:24:13	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-2xpx4	Started	Started container kube-rbac-proxy
20:24:18 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:24:18 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:24:52 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:24:52 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:25:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	InferenceServiceReady	InferenceService [isvc-paddle-runtime] is Ready
20:25:00 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:25:03 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:25:03 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:25:07	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:25:07	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:25:09	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-2xpx4	Killing	Stopping container kube-rbac-proxy
20:25:09	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-2xpx4	Killing	Stopping container kserve-container
20:25:10	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-2xpx4	Unhealthy	Readiness probe failed: Get "https://10.134.0.64:8643/healthz": dial tcp 10.134.0.64:8643: connect: connection refused
20:25:10 (x7)	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-2xpx4	Unhealthy	Readiness probe failed: dial tcp 10.134.0.64:8080: connect: connection refused
20:25:12 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:25:12	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-custom-pass] is Ready
20:25:14	kserve-ci-e2e-test	deployment-controller	isvc-paddle-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-v2-kserve-predictor-64d4c74b4d from 0 to 1
20:25:14	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-v2-kserve-predictor-64d4c74b4d	SuccessfulCreate	Created pod: isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt
20:25:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
20:25:14	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
20:25:15	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:25:15	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt	Started	Started container storage-initializer
20:25:15	kserve-ci-e2e-test	multus	isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt	AddedInterface	Add eth0 [10.134.0.65/23] from ovn-kubernetes
20:25:15	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt	Created	Created container: storage-initializer
20:25:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb	Killing	Stopping container kserve-container
20:25:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb	Killing	Stopping container kube-rbac-proxy
20:25:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-fail": the object has been modified; please apply your changes to the latest version and try again
20:25:19	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx
20:25:19	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6 from 0 to 1
20:25:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb	Unhealthy	Readiness probe failed: Get "https://10.134.0.63:8643/healthz": dial tcp 10.134.0.63:8643: connect: connection refused
20:25:20	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx	AddedInterface	Add eth0 [10.134.0.66/23] from ovn-kubernetes
20:25:20	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt	Pulled	Container image "kserve/paddleserver:latest" already present on machine
20:25:20	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt	Created	Created container: kserve-container
20:25:20	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt	Started	Started container kserve-container
20:25:20	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:25:20	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt	Created	Created container: kube-rbac-proxy
20:25:20	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt	Started	Started container kube-rbac-proxy
20:25:22 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb	Unhealthy	Readiness probe failed: dial tcp 10.134.0.63:8080: connect: connection refused
20:25:24 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx	Created	Created container: storage-initializer
20:25:24 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx	Started	Started container storage-initializer
20:25:24 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:25:29 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:25:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx	BackOff	Back-off restarting failed container storage-initializer in pod isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx_kserve-ci-e2e-test(71d8535a-24fa-4f82-aedc-4a959b539bf2)
20:25:30	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-pass": the object has been modified; please apply your changes to the latest version and try again
20:25:30	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689 from 0 to 1
20:25:30	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm
20:25:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm	Started	Started container storage-initializer
20:25:31	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm	AddedInterface	Add eth0 [10.134.0.67/23] from ovn-kubernetes
20:25:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:25:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm	Created	Created container: storage-initializer
20:25:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm	Created	Created container: kserve-container
20:25:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:25:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm	Started	Started container kube-rbac-proxy
20:25:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm	Created	Created container: kube-rbac-proxy
20:25:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:25:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm	Started	Started container kserve-container
20:25:44 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:25:44 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:26:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:26:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:26:07 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:26:07	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	InferenceServiceReady	InferenceService [isvc-paddle-v2-kserve] is Ready
20:26:14 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:26:14 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:26:16	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt	Killing	Stopping container kserve-container
20:26:16	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt	Killing	Stopping container kube-rbac-proxy
20:26:17	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt	Unhealthy	Readiness probe failed: Get "https://10.134.0.65:8643/healthz": dial tcp 10.134.0.65:8643: connect: connection refused
20:26:17 (x7)	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt	Unhealthy	Readiness probe failed: dial tcp 10.134.0.65:8080: connect: connection refused
20:26:21	kserve-ci-e2e-test	multus	isvc-pmml-predictor-8674795f59-6tpkp	AddedInterface	Add eth0 [10.134.0.68/23] from ovn-kubernetes
20:26:21	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-predictor-8674795f59	SuccessfulCreate	Created pod: isvc-pmml-predictor-8674795f59-6tpkp
20:26:21	kserve-ci-e2e-test	deployment-controller	isvc-pmml-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-predictor-8674795f59 from 0 to 1
20:26:21	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6tpkp	Created	Created container: storage-initializer
20:26:21	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6tpkp	Started	Started container storage-initializer
20:26:21	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6tpkp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:26:21	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	UpdateFailed	Failed to update status for InferenceService "isvc-pmml": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml": the object has been modified; please apply your changes to the latest version and try again
20:26:25	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6tpkp	Pulling	Pulling image "kserve/pmmlserver:latest"
20:26:33	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6tpkp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:26:33	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6tpkp	Pulled	Successfully pulled image "kserve/pmmlserver:latest" in 7.317s (7.317s including waiting). Image size: 853725565 bytes.
20:26:33	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6tpkp	Created	Created container: kserve-container
20:26:33	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6tpkp	Started	Started container kserve-container
20:26:33	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6tpkp	Started	Started container kube-rbac-proxy
20:26:33	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6tpkp	Created	Created container: kube-rbac-proxy
20:26:40 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm	Unhealthy	Readiness probe failed: dial tcp 10.134.0.67:8080: connect: connection refused
20:26:45 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:26:45 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:26:50 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:26:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm	Killing	Stopping container kube-rbac-proxy
20:26:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm	Killing	Stopping container kserve-container
20:26:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-serving-pass] is Ready
20:26:51	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5 from 0 to 1
20:26:51	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-fail": the object has been modified; please apply your changes to the latest version and try again
20:26:52	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n
20:26:52	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n	AddedInterface	Add eth0 [10.134.0.69/23] from ovn-kubernetes
20:26:56 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:26:57 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:26:57 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n	Created	Created container: storage-initializer
20:26:57 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n	Started	Started container storage-initializer
20:27:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n	Killing	Stopping container storage-initializer
20:27:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
20:27:03	kserve-ci-e2e-test	deployment-controller	isvc-pmml-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-v2-kserve-predictor-84cb55486c from 0 to 1
20:27:03	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
20:27:03	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-v2-kserve-predictor-84cb55486c	SuccessfulCreate	Created pod: isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz
20:27:04	kserve-ci-e2e-test	multus	isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz	AddedInterface	Add eth0 [10.134.0.70/23] from ovn-kubernetes
20:27:04	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:27:04	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz	Created	Created container: storage-initializer
20:27:04	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz	Started	Started container storage-initializer
20:27:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:27:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:27:09	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz	Started	Started container kube-rbac-proxy
20:27:09	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz	Created	Created container: kserve-container
20:27:09	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
20:27:09	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz	Started	Started container kserve-container
20:27:09	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz	Created	Created container: kube-rbac-proxy
20:27:09	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:27:18	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:27:18	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:27:50 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6tpkp	Unhealthy	Readiness probe failed: dial tcp 10.134.0.68:8080: connect: connection refused
20:27:51 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:27:51 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:28:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	InferenceServiceReady	InferenceService [isvc-pmml] is Ready
20:28:00 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:28:02	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6tpkp	Killing	Stopping container kube-rbac-proxy
20:28:02	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6tpkp	Killing	Stopping container kserve-container
20:28:05	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6tpkp	Unhealthy	Readiness probe failed: Get "https://10.134.0.68:8643/healthz": dial tcp 10.134.0.68:8643: connect: connection refused
20:28:07	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-runtime": the object has been modified; please apply your changes to the latest version and try again
20:28:07	kserve-ci-e2e-test	deployment-controller	isvc-pmml-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-runtime-predictor-885fc8777 from 0 to 1
20:28:07	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-runtime-predictor-885fc8777	SuccessfulCreate	Created pod: isvc-pmml-runtime-predictor-885fc8777-hvx2w
20:28:08	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-hvx2w	Created	Created container: storage-initializer
20:28:08	kserve-ci-e2e-test	multus	isvc-pmml-runtime-predictor-885fc8777-hvx2w	AddedInterface	Add eth0 [10.134.0.71/23] from ovn-kubernetes
20:28:08	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-hvx2w	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:28:08	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-hvx2w	Started	Started container storage-initializer
20:28:13	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-hvx2w	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
20:28:13	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-hvx2w	Created	Created container: kserve-container
20:28:13	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-hvx2w	Started	Started container kserve-container
20:28:13	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-hvx2w	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:28:13	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-hvx2w	Created	Created container: kube-rbac-proxy
20:28:13	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-hvx2w	Started	Started container kube-rbac-proxy
20:28:45	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	InferenceServiceReady	InferenceService [isvc-pmml-v2-kserve] is Ready
20:28:45 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:28:48 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:28:48 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:28:52 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:28:52 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:28:55	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz	Killing	Stopping container kube-rbac-proxy
20:28:55 (x12)	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz	Unhealthy	Readiness probe failed: dial tcp 10.134.0.70:8080: connect: connection refused
20:28:55	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz	Killing	Stopping container kserve-container
20:28:56	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz	Unhealthy	Readiness probe failed: Get "https://10.134.0.70:8643/healthz": dial tcp 10.134.0.70:8643: connect: connection refused
20:29:49 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:29:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	InferenceServiceReady	InferenceService [isvc-pmml-runtime] is Ready
20:29:52 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:29:52 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:29:59	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-hvx2w	Killing	Stopping container kserve-container
20:29:59 (x12)	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-hvx2w	Unhealthy	Readiness probe failed: dial tcp 10.134.0.71:8080: connect: connection refused
20:29:59	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-hvx2w	Killing	Stopping container kube-rbac-proxy
20:30:01	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-hvx2w	Unhealthy	Readiness probe failed: Get "https://10.134.0.71:8643/healthz": dial tcp 10.134.0.71:8643: connect: connection refused
20:30:04 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-274bef	UpdateFailed	Failed to update status for InferenceService "isvc-primary-274bef": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-274bef": the object has been modified; please apply your changes to the latest version and try again
20:30:04 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-274bef	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-274bef": the object has been modified; please apply your changes to the latest version and try again
20:30:04	kserve-ci-e2e-test	replicaset-controller	isvc-primary-274bef-predictor-57b4586b86	SuccessfulCreate	Created pod: isvc-primary-274bef-predictor-57b4586b86-zd9wv
20:30:04	kserve-ci-e2e-test	deployment-controller	isvc-primary-274bef-predictor	ScalingReplicaSet	Scaled up replica set isvc-primary-274bef-predictor-57b4586b86 from 0 to 1
20:30:05	kserve-ci-e2e-test	kubelet	isvc-primary-274bef-predictor-57b4586b86-zd9wv	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-primary-274bef-predictor-serving-cert" not found
20:30:06	kserve-ci-e2e-test	kubelet	isvc-primary-274bef-predictor-57b4586b86-zd9wv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:30:06	kserve-ci-e2e-test	kubelet	isvc-primary-274bef-predictor-57b4586b86-zd9wv	Started	Started container storage-initializer
20:30:06	kserve-ci-e2e-test	kubelet	isvc-primary-274bef-predictor-57b4586b86-zd9wv	Created	Created container: storage-initializer
20:30:06	kserve-ci-e2e-test	multus	isvc-primary-274bef-predictor-57b4586b86-zd9wv	AddedInterface	Add eth0 [10.134.0.72/23] from ovn-kubernetes
20:30:10	kserve-ci-e2e-test	kubelet	isvc-primary-274bef-predictor-57b4586b86-zd9wv	Started	Started container kube-rbac-proxy
20:30:10	kserve-ci-e2e-test	kubelet	isvc-primary-274bef-predictor-57b4586b86-zd9wv	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1907" already present on machine
20:30:10	kserve-ci-e2e-test	kubelet	isvc-primary-274bef-predictor-57b4586b86-zd9wv	Created	Created container: kserve-container
20:30:10	kserve-ci-e2e-test	kubelet	isvc-primary-274bef-predictor-57b4586b86-zd9wv	Started	Started container kserve-container
20:30:10	kserve-ci-e2e-test	kubelet	isvc-primary-274bef-predictor-57b4586b86-zd9wv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:30:10	kserve-ci-e2e-test	kubelet	isvc-primary-274bef-predictor-57b4586b86-zd9wv	Created	Created container: kube-rbac-proxy
20:30:19	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-274bef-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:30:19	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-274bef-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:31:18 (x9)	kserve-ci-e2e-test	kubelet	isvc-primary-274bef-predictor-57b4586b86-zd9wv	Unhealthy	Readiness probe failed: dial tcp 10.134.0.72:8080: connect: connection refused
20:31:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-274bef	InferenceServiceReady	InferenceService [isvc-primary-274bef] is Ready
20:31:28 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-274bef	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:31:49	kserve-ci-e2e-test	deployment-controller	isvc-secondary-274bef-predictor	ScalingReplicaSet	Scaled up replica set isvc-secondary-274bef-predictor-6665d79445 from 0 to 1
20:31:49 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-274bef-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:31:49 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-274bef-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:31:50 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-274bef	UpdateFailed	Failed to update status for InferenceService "isvc-secondary-274bef": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-274bef": the object has been modified; please apply your changes to the latest version and try again
20:31:50 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-274bef	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-274bef": the object has been modified; please apply your changes to the latest version and try again
20:31:50	kserve-ci-e2e-test	multus	isvc-secondary-274bef-predictor-6665d79445-nw7q9	AddedInterface	Add eth0 [10.134.0.73/23] from ovn-kubernetes
20:31:50	kserve-ci-e2e-test	replicaset-controller	isvc-secondary-274bef-predictor-6665d79445	SuccessfulCreate	Created pod: isvc-secondary-274bef-predictor-6665d79445-nw7q9
20:31:53 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-274bef-predictor-6665d79445-nw7q9	Created	Created container: storage-initializer
20:31:53 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-274bef-predictor-6665d79445-nw7q9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:1fa7c8d29012cdb239c749146593a8aa2685db50366be2fc7d147daf97857e63" already present on machine
20:31:53 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-274bef-predictor-6665d79445-nw7q9	Started	Started container storage-initializer
20:31:56 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-274bef	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:31:56	kserve-ci-e2e-test	kubelet	isvc-secondary-274bef-predictor-6665d79445-nw7q9	BackOff	Back-off restarting failed container storage-initializer in pod isvc-secondary-274bef-predictor-6665d79445-nw7q9_kserve-ci-e2e-test(3f0f002b-95a9-4e9a-abbf-661db6649a1f)
20:32:04	kserve-ci-e2e-test	kubelet	isvc-primary-274bef-predictor-57b4586b86-zd9wv	Killing	Stopping container kserve-container
20:32:04	kserve-ci-e2e-test	kubelet	isvc-primary-274bef-predictor-57b4586b86-zd9wv	Killing	Stopping container kube-rbac-proxy