Events

Time	Namespace	Component	RelatedObject	Reason	Message
19:18:56	kserve-ci-e2e-test		isvc-canary-stop-predictor-5ccc77f5f4-8l7nw	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw to ip-10-0-129-190.ec2.internal
19:29:45	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh to ip-10-0-129-50.ec2.internal
19:09:54	kserve-ci-e2e-test		isvc-canary-create-v2-predictor-85b98c9747-6cpd7	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-v2-predictor-85b98c9747-6cpd7 to ip-10-0-129-50.ec2.internal
18:57:04	kserve-ci-e2e-test		isvc-logger-predictor-5d8c64c8b4-9mcc7	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-logger-predictor-5d8c64c8b4-9mcc7 to ip-10-0-129-50.ec2.internal
19:14:44	kserve-ci-e2e-test		isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt to ip-10-0-129-50.ec2.internal
19:11:29	kserve-ci-e2e-test		isvc-canary-promote-predictor-6b6f66ff94-cn57m	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-cn57m to ip-10-0-129-50.ec2.internal
19:28:45	kserve-ci-e2e-test		isvc-paddle-runtime-predictor-58b99fff76-m4465	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465 to ip-10-0-129-190.ec2.internal
19:20:30	kserve-ci-e2e-test		xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv	Scheduled	Successfully assigned kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv to ip-10-0-129-50.ec2.internal
19:02:10	kserve-ci-e2e-test		isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4 to ip-10-0-129-50.ec2.internal
19:23:48	kserve-ci-e2e-test		isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln to ip-10-0-129-50.ec2.internal
19:25:08	kserve-ci-e2e-test		isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc to ip-10-0-129-50.ec2.internal
18:57:14	kserve-ci-e2e-test		isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p to ip-10-0-129-190.ec2.internal
19:13:58	kserve-ci-e2e-test		isvc-tensorflow-predictor-547dfc5669-xch6v	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v to ip-10-0-129-190.ec2.internal
19:09:42	kserve-ci-e2e-test		isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd to ip-10-0-129-50.ec2.internal
19:01:41	kserve-ci-e2e-test		isvc-autogluon-v2-predictor-9684b4cd9-ztppk	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk to ip-10-0-129-190.ec2.internal
19:11:29	kserve-ci-e2e-test		isvc-canary-promote-predictor-6b6f66ff94-kthkb	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb to ip-10-0-129-190.ec2.internal
19:26:03	kserve-ci-e2e-test		isvc-sklearn-mcp-predictor-76d4c69869-4jtdv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv to ip-10-0-129-190.ec2.internal
19:04:14	kserve-ci-e2e-test		isvc-sklearn-predictor-78d7749847-5xhdr	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-predictor-78d7749847-5xhdr to ip-10-0-129-50.ec2.internal
19:33:25	kserve-ci-e2e-test		isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt to ip-10-0-129-190.ec2.internal
19:21:25	kserve-ci-e2e-test		isvc-xgboost-runtime-predictor-54dd488c85-9gxq2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2 to ip-10-0-129-190.ec2.internal
19:19:34	kserve-ci-e2e-test		isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9 to ip-10-0-129-50.ec2.internal
19:08:26	kserve-ci-e2e-test		isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg to ip-10-0-129-190.ec2.internal
19:23:56	kserve-ci-e2e-test		isvc-xgboost-v2-predictor-fcc668bc-w5xtm	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm to ip-10-0-129-190.ec2.internal
19:22:50	kserve-ci-e2e-test		isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k to ip-10-0-129-50.ec2.internal
19:20:36	kserve-ci-e2e-test		isvc-lightgbm-predictor-54f85ddf88-djvhk	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk to ip-10-0-129-190.ec2.internal
19:30:01	kserve-ci-e2e-test		isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps to ip-10-0-129-190.ec2.internal
18:55:39	kserve-ci-e2e-test		isvc-predictive-xgboost-predictor-695f978978-4wjgg	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg to ip-10-0-129-190.ec2.internal
19:14:40	kserve-ci-e2e-test		isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs to ip-10-0-129-190.ec2.internal
19:00:45	kserve-ci-e2e-test		isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4 to ip-10-0-129-50.ec2.internal
19:14:40	kserve-ci-e2e-test		isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg to ip-10-0-129-190.ec2.internal
19:12:23	kserve-ci-e2e-test		isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j to ip-10-0-129-190.ec2.internal
19:09:54	kserve-ci-e2e-test		isvc-canary-create-predictor-76cc455874-skjsj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-skjsj to ip-10-0-129-50.ec2.internal
19:14:40	kserve-ci-e2e-test		isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl to ip-10-0-129-50.ec2.internal
19:32:54	kserve-ci-e2e-test		isvc-primary-63d815-predictor-54888c955d-rw7wt	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt to ip-10-0-129-190.ec2.internal
19:29:56	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t to ip-10-0-129-190.ec2.internal
19:31:27	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22 to ip-10-0-129-190.ec2.internal
18:53:24	kserve-ci-e2e-test		isvc-init-fail-28115f-predictor-57bb46966f-jx4sx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx to ip-10-0-129-190.ec2.internal
18:56:44	kserve-ci-e2e-test		message-dumper-predictor-6575bcfdfd-kmhd5	Scheduled	Successfully assigned kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5 to ip-10-0-129-190.ec2.internal
19:09:14	kserve-ci-e2e-test		isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt to ip-10-0-129-190.ec2.internal
19:18:55	kserve-ci-e2e-test		isvc-canary-stop-v2-predictor-854695c447-qpcx4	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4 to ip-10-0-129-190.ec2.internal
19:09:05	kserve-ci-e2e-test		isvc-sklearn-runtime-predictor-7f586df5dc-mdl94	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7f586df5dc-mdl94 to ip-10-0-129-50.ec2.internal
19:31:38	kserve-ci-e2e-test		isvc-pmml-runtime-predictor-885fc8777-scj7n	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n to ip-10-0-129-190.ec2.internal
19:25:21	kserve-ci-e2e-test		isvc-sklearn-s3-predictor-6786d494d6-9vddp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp to ip-10-0-129-190.ec2.internal
19:10:47	kserve-ci-e2e-test		isvc-sklearn-v2-predictor-866dcdbf79-5gnxs	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs to ip-10-0-129-190.ec2.internal
19:28:13	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c to ip-10-0-129-190.ec2.internal
18:58:30	kserve-ci-e2e-test		isvc-autogluon-v1-predictor-574b5b7495-s2xwx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx to ip-10-0-129-190.ec2.internal
19:05:49	kserve-ci-e2e-test		sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4	Scheduled	Successfully assigned kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4 to ip-10-0-129-50.ec2.internal
18:58:50	kserve-ci-e2e-test		isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79 to ip-10-0-129-50.ec2.internal
19:09:54	kserve-ci-e2e-test		isvc-canary-create-predictor-76cc455874-rktzp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp to ip-10-0-129-190.ec2.internal
18:55:09	kserve-ci-e2e-test		isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz to ip-10-0-129-50.ec2.internal
19:13:05	kserve-ci-e2e-test		isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj to ip-10-0-129-50.ec2.internal
19:34:40	kserve-ci-e2e-test		isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f to ip-10-0-129-50.ec2.internal
19:27:29	kserve-ci-e2e-test		isvc-paddle-predictor-6d5c9fd4f6-4fvsm	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4fvsm to ip-10-0-129-50.ec2.internal
19:17:59	kserve-ci-e2e-test		isvc-xgboost-predictor-7566d65bf7-gwfg9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9 to ip-10-0-129-190.ec2.internal
19:11:29	kserve-ci-e2e-test		isvc-canary-promote-v2-predictor-769cdc7dcc-prp88	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88 to ip-10-0-129-190.ec2.internal
19:24:43	kserve-ci-e2e-test		isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r to ip-10-0-129-190.ec2.internal
19:18:55	kserve-ci-e2e-test		isvc-canary-stop-predictor-5ccc77f5f4-fbb6q	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-fbb6q to ip-10-0-129-50.ec2.internal
19:22:12	kserve-ci-e2e-test		isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8 to ip-10-0-129-190.ec2.internal
19:15:31	kserve-ci-e2e-test		isvc-triton-predictor-5d7db58676-f8flm	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm to ip-10-0-129-190.ec2.internal
18:53:44	kserve-ci-e2e-test		isvc-predictive-sklearn-predictor-5f849d69df-hv478	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478 to ip-10-0-129-190.ec2.internal
19:03:33	kserve-ci-e2e-test		isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8 to ip-10-0-129-190.ec2.internal
19:16:20	kserve-ci-e2e-test		isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9 to ip-10-0-129-190.ec2.internal
18:53:24	kserve-ci-e2e-test		isvc-sklearn-batcher-predictor-598fd877f-xqq2c	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-598fd877f-xqq2c to ip-10-0-129-50.ec2.internal
19:26:41	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4 to ip-10-0-129-50.ec2.internal
19:31:08	kserve-ci-e2e-test		isvc-pmml-predictor-8674795f59-lmbkk	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk to ip-10-0-129-190.ec2.internal
19:28:02	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br to ip-10-0-129-190.ec2.internal
19:05:54	kserve-ci-e2e-test		isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55 to ip-10-0-129-190.ec2.internal
18:53:23	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-predictor-598fd877f from 0 to 1
18:53:24	kserve-ci-e2e-test	replicaset-controller	isvc-init-fail-28115f-predictor-57bb46966f	SuccessfulCreate	Created pod: isvc-init-fail-28115f-predictor-57bb46966f-jx4sx
18:53:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-598fd877f-xqq2c	Created	Created container: storage-initializer
18:53:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-598fd877f-xqq2c	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
18:53:24	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-predictor-598fd877f-xqq2c	AddedInterface	Add eth0 [10.133.0.29/23] from ovn-kubernetes
18:53:24	kserve-ci-e2e-test	kubelet	isvc-init-fail-28115f-predictor-57bb46966f-jx4sx	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-init-fail-28115f-predictor-serving-cert" not found
18:53:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-598fd877f-xqq2c	Started	Started container storage-initializer
18:53:24	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-predictor-598fd877f	SuccessfulCreate	Created pod: isvc-sklearn-batcher-predictor-598fd877f-xqq2c
18:53:24	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
18:53:24	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
18:53:24	kserve-ci-e2e-test	deployment-controller	isvc-init-fail-28115f-predictor	ScalingReplicaSet	Scaled up replica set isvc-init-fail-28115f-predictor-57bb46966f from 0 to 1
18:53:25	kserve-ci-e2e-test	multus	isvc-init-fail-28115f-predictor-57bb46966f-jx4sx	AddedInterface	Add eth0 [10.134.0.34/23] from ovn-kubernetes
18:53:25	kserve-ci-e2e-test	kubelet	isvc-init-fail-28115f-predictor-57bb46966f-jx4sx	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea"
18:53:28	kserve-ci-e2e-test	kubelet	isvc-init-fail-28115f-predictor-57bb46966f-jx4sx	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" in 3.324s (3.324s including waiting). Image size: 306757058 bytes.
18:53:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-598fd877f-xqq2c	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987"
18:53:35	kserve-ci-e2e-test	kubelet	isvc-init-fail-28115f-predictor-57bb46966f-jx4sx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
18:53:35 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-28115f-predictor-57bb46966f-jx4sx	Started	Started container storage-initializer
18:53:35 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-28115f-predictor-57bb46966f-jx4sx	Created	Created container: storage-initializer
18:53:38	kserve-ci-e2e-test	kubelet	isvc-init-fail-28115f-predictor-57bb46966f-jx4sx	BackOff	Back-off restarting failed container storage-initializer in pod isvc-init-fail-28115f-predictor-57bb46966f-jx4sx_kserve-ci-e2e-test(ae207379-aa2c-4b5e-b959-b4562ec3623e)
18:53:38 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-28115f	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:53:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-598fd877f-xqq2c	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
18:53:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-598fd877f-xqq2c	Started	Started container kserve-container
18:53:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-598fd877f-xqq2c	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" in 12.952s (12.952s including waiting). Image size: 1580716956 bytes.
18:53:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-598fd877f-xqq2c	Created	Created container: kserve-container
18:53:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-598fd877f-xqq2c	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.145s (2.145s including waiting). Image size: 211946088 bytes.
18:53:44	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-hv478	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
18:53:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-598fd877f-xqq2c	Created	Created container: kube-rbac-proxy
18:53:44	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-hv478	Created	Created container: storage-initializer
18:53:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-598fd877f-xqq2c	Pulling	Pulling image "quay.io/opendatahub/kserve-agent@sha256:b2f25d6795b8ebd41ededee75d11d5b4c26ab534501a041bcd9b37043528824a"
18:53:44	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-predictor-5f849d69df	SuccessfulCreate	Created pod: isvc-predictive-sklearn-predictor-5f849d69df-hv478
18:53:44	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-predictor-5f849d69df-hv478	AddedInterface	Add eth0 [10.134.0.35/23] from ovn-kubernetes
18:53:44	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-hv478	Started	Started container storage-initializer
18:53:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn": the object has been modified; please apply your changes to the latest version and try again
18:53:44	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-predictor-5f849d69df from 0 to 1
18:53:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-598fd877f-xqq2c	Started	Started container kube-rbac-proxy
18:53:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-598fd877f-xqq2c	Started	Started container agent
18:53:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-598fd877f-xqq2c	Created	Created container: agent
18:53:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-598fd877f-xqq2c	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:b2f25d6795b8ebd41ededee75d11d5b4c26ab534501a041bcd9b37043528824a" in 2.144s (2.144s including waiting). Image size: 239960677 bytes.
18:53:49	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-hv478	Pulling	Pulling image "kserve/predictiveserver:latest"
18:54:10	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-hv478	Pulled	Successfully pulled image "kserve/predictiveserver:latest" in 21.542s (21.542s including waiting). Image size: 2434679947 bytes.
18:54:11	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-hv478	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
18:54:11	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-hv478	Started	Started container kserve-container
18:54:11	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-hv478	Created	Created container: kserve-container
18:54:14	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-hv478	Created	Created container: kube-rbac-proxy
18:54:14	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-hv478	Started	Started container kube-rbac-proxy
18:54:14	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-hv478	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 3.241s (3.241s including waiting). Image size: 211946088 bytes.
18:54:24 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:54:24 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:54:44 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:54:44 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:54:54 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:54:54 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:54:54 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:54:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InferenceServiceReady	InferenceService [isvc-sklearn-batcher] is Ready
18:55:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz	Started	Started container storage-initializer
18:55:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-598fd877f-xqq2c	Killing	Stopping container agent
18:55:09	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-custom-predictor-dfc48c89	SuccessfulCreate	Created pod: isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz
18:55:09	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz	AddedInterface	Add eth0 [10.133.0.30/23] from ovn-kubernetes
18:55:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
18:55:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz	Created	Created container: storage-initializer
18:55:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-598fd877f-xqq2c	Killing	Stopping container kube-rbac-proxy
18:55:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-598fd877f-xqq2c	Killing	Stopping container kserve-container
18:55:09 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher-custom": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher-custom": the object has been modified; please apply your changes to the latest version and try again
18:55:09	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-custom-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-custom-predictor-dfc48c89 from 0 to 1
18:55:09	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher-custom": the object has been modified; please apply your changes to the latest version and try again
18:55:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:b2f25d6795b8ebd41ededee75d11d5b4c26ab534501a041bcd9b37043528824a" already present on machine
18:55:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz	Created	Created container: kserve-container
18:55:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" already present on machine
18:55:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz	Started	Started container kserve-container
18:55:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:55:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz	Created	Created container: kube-rbac-proxy
18:55:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz	Started	Started container kube-rbac-proxy
18:55:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz	Created	Created container: agent
18:55:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz	Started	Started container agent
18:55:22 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-hv478	Unhealthy	Readiness probe failed: dial tcp 10.134.0.35:8080: connect: connection refused
18:55:23 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-598fd877f-xqq2c	Unhealthy	Readiness probe failed: dial tcp 10.133.0.29:8080: connect: connection refused
18:55:23 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-598fd877f-xqq2c	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
18:55:28 (x4)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-598fd877f-xqq2c	Unhealthy	Readiness probe failed: Get "https://10.133.0.29:8643/healthz": dial tcp 10.133.0.29:8643: connect: connection refused
18:55:29 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:55:29 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:55:32	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	InferenceServiceReady	InferenceService [isvc-predictive-sklearn] is Ready
18:55:32 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:55:34	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-hv478	Killing	Stopping container kube-rbac-proxy
18:55:34	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-hv478	Killing	Stopping container kserve-container
18:55:35	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-hv478	Unhealthy	Readiness probe failed: Get "https://10.134.0.35:8643/healthz": dial tcp 10.134.0.35:8643: connect: connection refused
18:55:39	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-predictor-695f978978	SuccessfulCreate	Created pod: isvc-predictive-xgboost-predictor-695f978978-4wjgg
18:55:39	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-predictor-695f978978 from 0 to 1
18:55:39 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:55:39 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:55:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost": the object has been modified; please apply your changes to the latest version and try again
18:55:40	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-4wjgg	Started	Started container storage-initializer
18:55:40	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-predictor-695f978978-4wjgg	AddedInterface	Add eth0 [10.134.0.36/23] from ovn-kubernetes
18:55:40	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-4wjgg	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
18:55:40	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-4wjgg	Created	Created container: storage-initializer
18:55:44	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-4wjgg	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
18:55:45	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-4wjgg	Started	Started container kserve-container
18:55:45	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-4wjgg	Created	Created container: kube-rbac-proxy
18:55:45	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-4wjgg	Created	Created container: kserve-container
18:55:45	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-4wjgg	Started	Started container kube-rbac-proxy
18:55:45	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-4wjgg	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:56:24 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:56:24 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:56:31 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:56:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	InferenceServiceReady	InferenceService [isvc-sklearn-batcher-custom] is Ready
18:56:39 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:56:39 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:56:44	kserve-ci-e2e-test	deployment-controller	message-dumper-predictor	ScalingReplicaSet	Scaled up replica set message-dumper-predictor-6575bcfdfd from 0 to 1
18:56:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz	Killing	Stopping container kserve-container
18:56:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz	Killing	Stopping container agent
18:56:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz	Killing	Stopping container kube-rbac-proxy
18:56:44	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	UpdateFailed	Failed to update status for InferenceService "message-dumper": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper": the object has been modified; please apply your changes to the latest version and try again
18:56:44	kserve-ci-e2e-test	replicaset-controller	message-dumper-predictor-6575bcfdfd	SuccessfulCreate	Created pod: message-dumper-predictor-6575bcfdfd-kmhd5
18:56:45	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-kmhd5	Pulling	Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display"
18:56:45	kserve-ci-e2e-test	multus	message-dumper-predictor-6575bcfdfd-kmhd5	AddedInterface	Add eth0 [10.134.0.37/23] from ovn-kubernetes
18:56:46	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-kmhd5	Started	Started container kube-rbac-proxy
18:56:46	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-kmhd5	Pulled	Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.119s (1.119s including waiting). Image size: 14813193 bytes.
18:56:46	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-kmhd5	Created	Created container: kube-rbac-proxy
18:56:46	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-kmhd5	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:56:46	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-kmhd5	Started	Started container kserve-container
18:56:46	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-kmhd5	Created	Created container: kserve-container
18:56:51 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
18:56:54 (x6)	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:56:54 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:56:54	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	InferenceServiceReady	InferenceService [message-dumper] is Ready
18:56:54 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:57:01 (x4)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz	Unhealthy	Readiness probe failed: Get "https://10.133.0.30:8643/healthz": dial tcp 10.133.0.30:8643: connect: connection refused
18:57:01 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-dfc48c89-j4mbz	Unhealthy	Readiness probe failed: dial tcp 10.133.0.30:5000: connect: connection refused
18:57:02 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:57:02	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	InferenceServiceReady	InferenceService [isvc-predictive-xgboost] is Ready
18:57:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	UpdateFailed	Failed to update status for InferenceService "isvc-logger": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger": the object has been modified; please apply your changes to the latest version and try again
18:57:04	kserve-ci-e2e-test	deployment-controller	isvc-logger-predictor	ScalingReplicaSet	Scaled up replica set isvc-logger-predictor-5d8c64c8b4 from 0 to 1
18:57:04	kserve-ci-e2e-test	replicaset-controller	isvc-logger-predictor-5d8c64c8b4	SuccessfulCreate	Created pod: isvc-logger-predictor-5d8c64c8b4-9mcc7
18:57:05	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5d8c64c8b4-9mcc7	Created	Created container: storage-initializer
18:57:05	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5d8c64c8b4-9mcc7	Started	Started container storage-initializer
18:57:05	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5d8c64c8b4-9mcc7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
18:57:05	kserve-ci-e2e-test	multus	isvc-logger-predictor-5d8c64c8b4-9mcc7	AddedInterface	Add eth0 [10.133.0.31/23] from ovn-kubernetes
18:57:09	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-4wjgg	Killing	Stopping container kube-rbac-proxy
18:57:09	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5d8c64c8b4-9mcc7	Created	Created container: kube-rbac-proxy
18:57:09	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetScale	deployments.apps "isvc-predictive-xgboost-predictor" not found
18:57:09	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5d8c64c8b4-9mcc7	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" already present on machine
18:57:09	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5d8c64c8b4-9mcc7	Created	Created container: kserve-container
18:57:09	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5d8c64c8b4-9mcc7	Started	Started container kserve-container
18:57:09	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-4wjgg	Killing	Stopping container kserve-container
18:57:09	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5d8c64c8b4-9mcc7	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:57:09	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5d8c64c8b4-9mcc7	Started	Started container kube-rbac-proxy
18:57:09	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5d8c64c8b4-9mcc7	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:b2f25d6795b8ebd41ededee75d11d5b4c26ab534501a041bcd9b37043528824a" already present on machine
18:57:10	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5d8c64c8b4-9mcc7	Created	Created container: agent
18:57:10	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5d8c64c8b4-9mcc7	Started	Started container agent
18:57:11	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-4wjgg	Unhealthy	Readiness probe failed: Get "https://10.134.0.36:8643/healthz": dial tcp 10.134.0.36:8643: connect: connection refused
18:57:11 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-4wjgg	Unhealthy	Readiness probe failed: dial tcp 10.134.0.36:8080: connect: connection refused
18:57:14 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:57:14	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-predictor-ff56bcbb5 from 0 to 1
18:57:14	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-predictor-ff56bcbb5	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p
18:57:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm": the object has been modified; please apply your changes to the latest version and try again
18:57:14 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:57:15	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
18:57:15	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
18:57:16	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p	Created	Created container: storage-initializer
18:57:16	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
18:57:16	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p	Started	Started container storage-initializer
18:57:20	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p	Created	Created container: kserve-container
18:57:20	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p	Started	Started container kube-rbac-proxy
18:57:20	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p	Started	Started container kserve-container
18:57:20	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
18:57:20	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:57:20	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p	Created	Created container: kube-rbac-proxy
18:57:34 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:57:34 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:57:59 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:57:59 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:58:19 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:58:19 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:58:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	InferenceServiceReady	InferenceService [isvc-logger] is Ready
18:58:22 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:58:29	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v1-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v1-predictor-574b5b7495 from 0 to 1
18:58:29	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-kmhd5	Killing	Stopping container kube-rbac-proxy
18:58:29	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-kmhd5	Killing	Stopping container kserve-container
18:58:29	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5d8c64c8b4-9mcc7	Killing	Stopping container kserve-container
18:58:29	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5d8c64c8b4-9mcc7	Killing	Stopping container agent
18:58:29	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5d8c64c8b4-9mcc7	Killing	Stopping container kube-rbac-proxy
18:58:30	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-s2xwx	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-autogluon-v1-predictor-serving-cert" not found
18:58:30	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v1-predictor-574b5b7495	SuccessfulCreate	Created pod: isvc-autogluon-v1-predictor-574b5b7495-s2xwx
18:58:31	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-s2xwx	Created	Created container: storage-initializer
18:58:31	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-s2xwx	Started	Started container storage-initializer
18:58:31	kserve-ci-e2e-test	multus	isvc-autogluon-v1-predictor-574b5b7495-s2xwx	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
18:58:31	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-s2xwx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
18:58:37 (x2)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5d8c64c8b4-9mcc7	Unhealthy	Readiness probe failed: Get "https://10.133.0.31:8643/healthz": dial tcp 10.133.0.31:8643: connect: connection refused
18:58:38 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:58:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm] is Ready
18:58:42 (x11)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5d8c64c8b4-9mcc7	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
18:58:42 (x12)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5d8c64c8b4-9mcc7	Unhealthy	Readiness probe failed: dial tcp 10.133.0.31:8080: connect: connection refused
18:58:44	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p	Killing	Stopping container kserve-container
18:58:44 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:58:44	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p	Killing	Stopping container kube-rbac-proxy
18:58:44 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:58:48	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p	Unhealthy	Readiness probe failed: Get "https://10.134.0.38:8643/healthz": dial tcp 10.134.0.38:8643: connect: connection refused
18:58:48 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p	Unhealthy	Readiness probe failed: dial tcp 10.134.0.38:8080: connect: connection refused
18:58:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
18:58:50	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
18:58:50	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-v2-predictor-745f9c8c9f	SuccessfulCreate	Created pod: isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79
18:58:50	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-v2-predictor-745f9c8c9f from 0 to 1
18:58:51	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79	Created	Created container: storage-initializer
18:58:51	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79	Started	Started container storage-initializer
18:58:51	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79	AddedInterface	Add eth0 [10.133.0.32/23] from ovn-kubernetes
18:58:51	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
18:58:55	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79	Pulling	Pulling image "kserve/predictiveserver:latest"
18:59:00	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-s2xwx	Pulling	Pulling image "kserve/autogluonserver:latest"
18:59:16	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79	Started	Started container kserve-container
18:59:16	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:59:16	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79	Created	Created container: kube-rbac-proxy
18:59:16	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79	Started	Started container kube-rbac-proxy
18:59:16	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79	Created	Created container: kserve-container
18:59:16	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79	Pulled	Successfully pulled image "kserve/predictiveserver:latest" in 21.23s (21.23s including waiting). Image size: 2434679947 bytes.
18:59:50 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:59:50 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:00:33 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:00:33	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	InferenceServiceReady	InferenceService [isvc-predictive-sklearn-v2] is Ready
19:00:35 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:00:35 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:00:40	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79	Killing	Stopping container kserve-container
19:00:40	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79	Killing	Stopping container kube-rbac-proxy
19:00:43 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79	Unhealthy	Readiness probe failed: Get "http://10.133.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready": dial tcp 10.133.0.32:8080: connect: connection refused
19:00:43	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-5hz79	Unhealthy	Readiness probe failed: Get "https://10.133.0.32:8643/healthz": dial tcp 10.133.0.32:8643: connect: connection refused
19:00:45	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-v2-predictor-868ff5dd6d	SuccessfulCreate	Created pod: isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4
19:00:45	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-v2-predictor-868ff5dd6d from 0 to 1
19:00:45	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
19:00:46	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4	AddedInterface	Add eth0 [10.133.0.33/23] from ovn-kubernetes
19:00:46	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:00:46	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4	Created	Created container: storage-initializer
19:00:46	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4	Started	Started container storage-initializer
19:00:50	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
19:00:51	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4	Started	Started container kube-rbac-proxy
19:00:51	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4	Created	Created container: kube-rbac-proxy
19:00:51	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4	Created	Created container: kserve-container
19:00:51	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4	Started	Started container kserve-container
19:00:51	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:01:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-s2xwx	Started	Started container kube-rbac-proxy
19:01:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-s2xwx	Created	Created container: kube-rbac-proxy
19:01:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-s2xwx	Started	Started container kserve-container
19:01:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-s2xwx	Created	Created container: kserve-container
19:01:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-s2xwx	Pulled	Successfully pulled image "kserve/autogluonserver:latest" in 2m2.182s (2m2.182s including waiting). Image size: 8945772325 bytes.
19:01:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-s2xwx	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:01:10 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-s2xwx	Unhealthy	Readiness probe failed: dial tcp 10.134.0.39:8080: connect: connection refused
19:01:20 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:01:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	InferenceServiceReady	InferenceService [isvc-autogluon-v1] is Ready
19:01:30 (x12)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:01:30 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:01:30 (x12)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:01:30 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:01:36	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-s2xwx	Killing	Stopping container kube-rbac-proxy
19:01:36	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-s2xwx	Killing	Stopping container kserve-container
19:01:41	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-predictor-9684b4cd9 from 0 to 1
19:01:41	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-predictor-9684b4cd9	SuccessfulCreate	Created pod: isvc-autogluon-v2-predictor-9684b4cd9-ztppk
19:01:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2": the object has been modified; please apply your changes to the latest version and try again
19:01:42	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-ztppk	Started	Started container storage-initializer
19:01:42	kserve-ci-e2e-test	multus	isvc-autogluon-v2-predictor-9684b4cd9-ztppk	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
19:01:42	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-ztppk	Created	Created container: storage-initializer
19:01:42	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-ztppk	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:01:47	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-ztppk	Created	Created container: kube-rbac-proxy
19:01:47	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-ztppk	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:01:47	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-ztppk	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
19:01:47	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-ztppk	Created	Created container: kserve-container
19:01:47	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-ztppk	Started	Started container kube-rbac-proxy
19:01:47	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-ztppk	Started	Started container kserve-container
19:01:56	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:01:56	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:01:59 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:01:59	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	InferenceServiceReady	InferenceService [isvc-predictive-xgboost-v2] is Ready
19:02:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:02:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:02:05	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4	Killing	Stopping container kserve-container
19:02:05	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4	Killing	Stopping container kube-rbac-proxy
19:02:08	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4	Unhealthy	Readiness probe failed: Get "https://10.133.0.33:8643/healthz": dial tcp 10.133.0.33:8643: connect: connection refused
19:02:08 (x4)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcdc4	Unhealthy	Readiness probe failed: Get "http://10.133.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready": dial tcp 10.133.0.33:8080: connect: connection refused
19:02:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm-v2": the object has been modified; please apply your changes to the latest version and try again
19:02:10	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-v2-predictor-545b7cc587 from 0 to 1
19:02:10	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-v2-predictor-545b7cc587	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4
19:02:11	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4	AddedInterface	Add eth0 [10.133.0.34/23] from ovn-kubernetes
19:02:11	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
19:02:12	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4	Started	Started container storage-initializer
19:02:12	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4	Created	Created container: storage-initializer
19:02:12	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:02:16	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
19:02:16	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4	Created	Created container: kube-rbac-proxy
19:02:16	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4	Started	Started container kube-rbac-proxy
19:02:16	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:02:16	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4	Started	Started container kserve-container
19:02:16	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4	Created	Created container: kserve-container
19:02:40 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:02:40 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:03:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	InferenceServiceReady	InferenceService [isvc-autogluon-v2] is Ready
19:03:20 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:03:25 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:03:25 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:03:26 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:03:26 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:03:27	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-ztppk	Killing	Stopping container kserve-container
19:03:27	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-ztppk	Killing	Stopping container kube-rbac-proxy
19:03:30 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:03:30	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm-v2] is Ready
19:03:33	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-variants-predictor-5c889d8844	SuccessfulCreate	Created pod: isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8
19:03:33	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-variants-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-variants-predictor-5c889d8844 from 0 to 1
19:03:33	kserve-ci-e2e-test	multus	isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8	AddedInterface	Add eth0 [10.134.0.41/23] from ovn-kubernetes
19:03:33	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:03:33	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8	Created	Created container: storage-initializer
19:03:33	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8	Started	Started container storage-initializer
19:03:33	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2-variants": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-variants": the object has been modified; please apply your changes to the latest version and try again
19:03:34	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4	Killing	Stopping container kserve-container
19:03:34	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4	Killing	Stopping container kube-rbac-proxy
19:03:39	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4	Unhealthy	Readiness probe failed: Get "https://10.133.0.34:8643/healthz": dial tcp 10.133.0.34:8643: connect: connection refused
19:03:40	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-scheduler-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-scheduler-predictor-57cfcc6855 from 0 to 1
19:03:40 (x4)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:03:40 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
19:03:40 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-scheduler": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
19:03:40	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-scheduler-predictor-57cfcc6855	SuccessfulCreate	Created pod: isvc-sklearn-scheduler-predictor-57cfcc6855-gzzhn
19:03:40 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-wb5k4	Unhealthy	Readiness probe failed: Get "http://10.133.0.34:8080/v2/models/isvc-predictive-lightgbm-v2/ready": dial tcp 10.133.0.34:8080: connect: connection refused
19:04:06	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8	Started	Started container kube-rbac-proxy
19:04:06	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
19:04:06	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:04:06	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8	Started	Started container kserve-container
19:04:06	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8	Created	Created container: kube-rbac-proxy
19:04:06	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8	Created	Created container: kserve-container
19:04:14	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-predictor-78d7749847 from 0 to 1
19:04:14	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-predictor-78d7749847	SuccessfulCreate	Created pod: isvc-sklearn-predictor-78d7749847-5xhdr
19:04:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn": the object has been modified; please apply your changes to the latest version and try again
19:04:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-78d7749847-5xhdr	Created	Created container: storage-initializer
19:04:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-78d7749847-5xhdr	Started	Started container storage-initializer
19:04:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-78d7749847-5xhdr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:04:15	kserve-ci-e2e-test	multus	isvc-sklearn-predictor-78d7749847-5xhdr	AddedInterface	Add eth0 [10.133.0.37/23] from ovn-kubernetes
19:04:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-78d7749847-5xhdr	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" already present on machine
19:04:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-78d7749847-5xhdr	Created	Created container: kserve-container
19:04:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-78d7749847-5xhdr	Started	Started container kube-rbac-proxy
19:04:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-78d7749847-5xhdr	Started	Started container kserve-container
19:04:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-78d7749847-5xhdr	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:04:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-78d7749847-5xhdr	Created	Created container: kube-rbac-proxy
19:04:48 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:04:48 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:04:59 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:04:59 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:05:36 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:05:36	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	InferenceServiceReady	InferenceService [isvc-autogluon-v2-variants] is Ready
19:05:38 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:05:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	InferenceServiceReady	InferenceService [isvc-sklearn] is Ready
19:05:44 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:05:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-78d7749847-5xhdr	Killing	Stopping container kube-rbac-proxy
19:05:44 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:05:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-78d7749847-5xhdr	Killing	Stopping container kserve-container
19:05:47	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8	Unhealthy	Readiness probe failed: Get "http://10.134.0.41:8080/v2/models/isvc-autogluon-v2-variants/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
19:05:47 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-78d7749847-5xhdr	Unhealthy	Readiness probe failed: dial tcp 10.133.0.37:8080: connect: connection refused
19:05:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-78d7749847-5xhdr	Unhealthy	Readiness probe failed: Get "https://10.133.0.37:8643/healthz": dial tcp 10.133.0.37:8643: connect: connection refused
19:05:48 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:05:48 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:05:49	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "sklearn-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
19:05:49	kserve-ci-e2e-test	deployment-controller	sklearn-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set sklearn-v2-mlserver-predictor-7f568cdc6b from 0 to 1
19:05:49	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8	Killing	Stopping container kube-rbac-proxy
19:05:49	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8	Killing	Stopping container kserve-container
19:05:49 (x2)	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "sklearn-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "sklearn-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
19:05:49	kserve-ci-e2e-test	replicaset-controller	sklearn-v2-mlserver-predictor-7f568cdc6b	SuccessfulCreate	Created pod: sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4
19:05:50	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:05:50	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4	Created	Created container: storage-initializer
19:05:50	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4	Started	Started container storage-initializer
19:05:50	kserve-ci-e2e-test	multus	sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4	AddedInterface	Add eth0 [10.133.0.38/23] from ovn-kubernetes
19:05:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2-noslash": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-noslash": the object has been modified; please apply your changes to the latest version and try again
19:05:54	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-noslash-predictor-5499c79f9f	SuccessfulCreate	Created pod: isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55
19:05:54	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-noslash-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-noslash-predictor-5499c79f9f from 0 to 1
19:05:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-noslash": the object has been modified; please apply your changes to the latest version and try again
19:05:54	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4	Pulling	Pulling image "docker.io/seldonio/mlserver:1.7.1"
19:05:55	kserve-ci-e2e-test	multus	isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55	AddedInterface	Add eth0 [10.134.0.42/23] from ovn-kubernetes
19:05:55	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55	Started	Started container storage-initializer
19:05:55	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55	Created	Created container: storage-initializer
19:05:55	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:06:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55	Started	Started container kserve-container
19:06:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55	Created	Created container: kserve-container
19:06:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
19:06:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:06:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55	Created	Created container: kube-rbac-proxy
19:06:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55	Started	Started container kube-rbac-proxy
19:06:54 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:06:54 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:08:05	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4	Pulled	Successfully pulled image "docker.io/seldonio/mlserver:1.7.1" in 2m11.565s (2m11.565s including waiting). Image size: 10890461297 bytes.
19:08:06	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	InferenceServiceReady	InferenceService [isvc-autogluon-v2-noslash] is Ready
19:08:06 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:08:06	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4	Created	Created container: kserve-container
19:08:06	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4	Started	Started container kserve-container
19:08:06	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:08:06	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4	Created	Created container: kube-rbac-proxy
19:08:06	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4	Started	Started container kube-rbac-proxy
19:08:10 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:08:10 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:08:17	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55	Unhealthy	Readiness probe failed: Get "http://10.134.0.42:8080/v2/models/isvc-autogluon-v2-noslash/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
19:08:21	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55	Killing	Stopping container kserve-container
19:08:21	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55	Killing	Stopping container kube-rbac-proxy
19:08:26	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
19:08:26	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1": the object has been modified; please apply your changes to the latest version and try again
19:08:26	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-ts-v1-predictor-6cf79d9f7d	SuccessfulCreate	Created pod: isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg
19:08:26 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-ts-v1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1": the object has been modified; please apply your changes to the latest version and try again
19:08:26	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-ts-v1-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-ts-v1-predictor-6cf79d9f7d from 0 to 1
19:08:27	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg	Created	Created container: storage-initializer
19:08:27	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg	Started	Started container storage-initializer
19:08:27	kserve-ci-e2e-test	multus	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg	AddedInterface	Add eth0 [10.134.0.43/23] from ovn-kubernetes
19:08:27	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:08:32	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg	Created	Created container: kube-rbac-proxy
19:08:32	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
19:08:32	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg	Created	Created container: kserve-container
19:08:32	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg	Started	Started container kserve-container
19:08:32	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:08:32	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg	Started	Started container kube-rbac-proxy
19:08:35 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:08:35 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:08:43	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
19:08:49 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:08:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	InferenceServiceReady	InferenceService [isvc-autogluon-ts-v1] is Ready
19:08:50	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:08:50	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:08:53 (x11)	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:08:53	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	InferenceServiceReady	InferenceService [sklearn-v2-mlserver] is Ready
19:08:56 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:08:56 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:09:00	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4	Killing	Stopping container kserve-container
19:09:00	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4	Killing	Stopping container kube-rbac-proxy
19:09:03	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4	Unhealthy	Readiness probe failed: Get "https://10.133.0.38:8643/healthz": dial tcp 10.133.0.38:8643: connect: connection refused
19:09:03	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-52hx4	Unhealthy	Readiness probe failed: Get "http://10.133.0.38:8080/v2/models/sklearn-v2-mlserver/ready": dial tcp 10.133.0.38:8080: connect: connection refused
19:09:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-7f586df5dc-mdl94	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
19:09:05	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-runtime": the object has been modified; please apply your changes to the latest version and try again
19:09:05	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-runtime-predictor-7f586df5dc from 0 to 1
19:09:05	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-runtime-predictor-7f586df5dc	SuccessfulCreate	Created pod: isvc-sklearn-runtime-predictor-7f586df5dc-mdl94
19:09:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-7f586df5dc-mdl94	Started	Started container storage-initializer
19:09:06	kserve-ci-e2e-test	multus	isvc-sklearn-runtime-predictor-7f586df5dc-mdl94	AddedInterface	Add eth0 [10.133.0.39/23] from ovn-kubernetes
19:09:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-7f586df5dc-mdl94	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:09:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-7f586df5dc-mdl94	Created	Created container: storage-initializer
19:09:08	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg	Killing	Stopping container kserve-container
19:09:08	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg	Killing	Stopping container kube-rbac-proxy
19:09:09 (x4)	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg	Unhealthy	Readiness probe failed: dial tcp 10.134.0.43:8080: connect: connection refused
19:09:09	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg	Unhealthy	Readiness probe failed: Get "https://10.134.0.43:8643/healthz": dial tcp 10.134.0.43:8643: connect: connection refused
19:09:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-7f586df5dc-mdl94	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:09:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-7f586df5dc-mdl94	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" already present on machine
19:09:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-7f586df5dc-mdl94	Created	Created container: kserve-container
19:09:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-7f586df5dc-mdl94	Started	Started container kserve-container
19:09:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-7f586df5dc-mdl94	Started	Started container kube-rbac-proxy
19:09:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-7f586df5dc-mdl94	Created	Created container: kube-rbac-proxy
19:09:13	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-ts-v1-noslash-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-ts-v1-noslash-predictor-b54fbf585 from 0 to 1
19:09:13	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-ts-v1-noslash": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1-noslash": the object has been modified; please apply your changes to the latest version and try again
19:09:14	kserve-ci-e2e-test	multus	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt	AddedInterface	Add eth0 [10.134.0.44/23] from ovn-kubernetes
19:09:14	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:09:14	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt	Started	Started container storage-initializer
19:09:14	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt	Created	Created container: storage-initializer
19:09:14	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585	SuccessfulCreate	Created pod: isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt
19:09:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
19:09:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:09:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt	Started	Started container kserve-container
19:09:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt	Created	Created container: kserve-container
19:09:20 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-7f586df5dc-mdl94	Unhealthy	Readiness probe failed: dial tcp 10.133.0.39:8080: connect: connection refused
19:09:20	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt	Started	Started container kube-rbac-proxy
19:09:20	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt	Created	Created container: kube-rbac-proxy
19:09:26 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt	Unhealthy	Readiness probe failed: dial tcp 10.134.0.44:8080: connect: connection refused
19:09:30	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-runtime] is Ready
19:09:30 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:09:35 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:09:35 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:09:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-7f586df5dc-mdl94	Killing	Stopping container kserve-container
19:09:37 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:09:37	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	InferenceServiceReady	InferenceService [isvc-autogluon-ts-v1-noslash] is Ready
19:09:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-7f586df5dc-mdl94	Killing	Stopping container kube-rbac-proxy
19:09:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
19:09:42	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-runtime-predictor-5f69c7d498	SuccessfulCreate	Created pod: isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd
19:09:42	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-runtime-predictor-5f69c7d498 from 0 to 1
19:09:43	kserve-ci-e2e-test	multus	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd	AddedInterface	Add eth0 [10.133.0.40/23] from ovn-kubernetes
19:09:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:09:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd	Created	Created container: storage-initializer
19:09:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd	Started	Started container storage-initializer
19:09:44 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:09:44 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:09:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:09:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd	Started	Started container kserve-container
19:09:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:09:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd	Created	Created container: kube-rbac-proxy
19:09:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd	Created	Created container: kserve-container
19:09:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd	Started	Started container kube-rbac-proxy
19:09:49	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt	Killing	Stopping container kserve-container
19:09:49	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt	Killing	Stopping container kube-rbac-proxy
19:09:54	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-create-v2-predictor-85b98c9747 from 0 to 1
19:09:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again
19:09:54	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-create-predictor-76cc455874 from 0 to 2
19:09:54	kserve-ci-e2e-test	multus	isvc-canary-create-predictor-76cc455874-skjsj	AddedInterface	Add eth0 [10.133.0.42/23] from ovn-kubernetes
19:09:54	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-v2-predictor-85b98c9747	SuccessfulCreate	Created pod: isvc-canary-create-v2-predictor-85b98c9747-6cpd7
19:09:54	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-76cc455874	SuccessfulCreate	Created pod: isvc-canary-create-predictor-76cc455874-rktzp
19:09:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	UpdateFailed	Failed to update status for InferenceService "isvc-canary-create": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again
19:09:54	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-76cc455874	SuccessfulCreate	Created pod: isvc-canary-create-predictor-76cc455874-skjsj
19:09:55	kserve-ci-e2e-test	multus	isvc-canary-create-predictor-76cc455874-rktzp	AddedInterface	Add eth0 [10.134.0.45/23] from ovn-kubernetes
19:09:55	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-rktzp	Started	Started container storage-initializer
19:09:55	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-skjsj	Created	Created container: storage-initializer
19:09:55	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-skjsj	Started	Started container storage-initializer
19:09:55	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-rktzp	Created	Created container: storage-initializer
19:09:55	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-rktzp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:09:55 (x2)	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-85b98c9747-6cpd7	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-create-v2-predictor-serving-cert" not found
19:09:55	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-skjsj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:09:56	kserve-ci-e2e-test	multus	isvc-canary-create-v2-predictor-85b98c9747-6cpd7	AddedInterface	Add eth0 [10.133.0.41/23] from ovn-kubernetes
19:09:56	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-85b98c9747-6cpd7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:09:56	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-85b98c9747-6cpd7	Created	Created container: storage-initializer
19:09:56	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-85b98c9747-6cpd7	Started	Started container storage-initializer
19:09:57	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:09:57	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:09:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-skjsj	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" already present on machine
19:09:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-rktzp	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987"
19:09:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-skjsj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:09:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-skjsj	Created	Created container: kserve-container
19:09:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-skjsj	Started	Started container kserve-container
19:09:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-skjsj	Started	Started container kube-rbac-proxy
19:09:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-skjsj	Created	Created container: kube-rbac-proxy
19:10:00	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-85b98c9747-6cpd7	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" already present on machine
19:10:00	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-85b98c9747-6cpd7	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:10:00	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-85b98c9747-6cpd7	Created	Created container: kserve-container
19:10:00	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-85b98c9747-6cpd7	Started	Started container kserve-container
19:10:00	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-85b98c9747-6cpd7	Started	Started container kube-rbac-proxy
19:10:00	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-85b98c9747-6cpd7	Created	Created container: kube-rbac-proxy
19:10:12	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-rktzp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:10:12	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-rktzp	Created	Created container: kserve-container
19:10:12	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-rktzp	Started	Started container kube-rbac-proxy
19:10:12	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-rktzp	Created	Created container: kube-rbac-proxy
19:10:12	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-rktzp	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" in 13.593s (13.593s including waiting). Image size: 1580716956 bytes.
19:10:12	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-rktzp	Started	Started container kserve-container
19:10:25	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
19:10:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-v2-runtime] is Ready
19:10:35 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:10:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd	Killing	Stopping container kserve-container
19:10:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd	Killing	Stopping container kube-rbac-proxy
19:10:42 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:10:42 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:10:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd	Unhealthy	Readiness probe failed: Get "https://10.133.0.40:8643/healthz": dial tcp 10.133.0.40:8643: connect: connection refused
19:10:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-9s5vd	Unhealthy	Readiness probe failed: Get "http://10.133.0.40:8080/v2/models/isvc-sklearn-v2-runtime/ready": dial tcp 10.133.0.40:8080: connect: connection refused
19:10:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
19:10:47	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-predictor-866dcdbf79 from 0 to 1
19:10:47	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-predictor-866dcdbf79	SuccessfulCreate	Created pod: isvc-sklearn-v2-predictor-866dcdbf79-5gnxs
19:10:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-866dcdbf79-5gnxs	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-v2-predictor-serving-cert" not found
19:10:49	kserve-ci-e2e-test	multus	isvc-sklearn-v2-predictor-866dcdbf79-5gnxs	AddedInterface	Add eth0 [10.134.0.46/23] from ovn-kubernetes
19:10:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-866dcdbf79-5gnxs	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:10:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-866dcdbf79-5gnxs	Created	Created container: storage-initializer
19:10:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-866dcdbf79-5gnxs	Started	Started container storage-initializer
19:10:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-866dcdbf79-5gnxs	Created	Created container: kserve-container
19:10:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-866dcdbf79-5gnxs	Created	Created container: kube-rbac-proxy
19:10:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-866dcdbf79-5gnxs	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" already present on machine
19:10:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-866dcdbf79-5gnxs	Started	Started container kserve-container
19:10:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-866dcdbf79-5gnxs	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:10:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-866dcdbf79-5gnxs	Started	Started container kube-rbac-proxy
19:11:08 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-85b98c9747-6cpd7	Unhealthy	Readiness probe failed: dial tcp 10.133.0.41:8080: connect: connection refused
19:11:18	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-76cc455874	SuccessfulDelete	Deleted pod: isvc-canary-create-predictor-76cc455874-rktzp
19:11:18	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-create-predictor-76cc455874 from 2 to 1
19:11:18	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-rktzp	Killing	Stopping container kube-rbac-proxy
19:11:18 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-rktzp	Unhealthy	Readiness probe failed: dial tcp 10.134.0.45:8080: connect: connection refused
19:11:18	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-rktzp	Killing	Stopping container kserve-container
19:11:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	InferenceServiceReady	InferenceService [isvc-canary-create] is Ready
19:11:19 (x20)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:11:24	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-85b98c9747-6cpd7	Killing	Stopping container kube-rbac-proxy
19:11:24	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-skjsj	Killing	Stopping container kube-rbac-proxy
19:11:24	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-skjsj	Killing	Stopping container kserve-container
19:11:24	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-85b98c9747-6cpd7	Killing	Stopping container kserve-container
19:11:27 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-skjsj	Unhealthy	Readiness probe failed: dial tcp 10.133.0.42:8080: connect: connection refused
19:11:27	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-skjsj	Unhealthy	Readiness probe failed: Get "https://10.133.0.42:8643/healthz": dial tcp 10.133.0.42:8643: connect: connection refused
19:11:29	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-6b6f66ff94	SuccessfulCreate	Created pod: isvc-canary-promote-predictor-6b6f66ff94-cn57m
19:11:29	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-v2-predictor-769cdc7dcc from 0 to 1
19:11:29	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-v2-predictor-769cdc7dcc	SuccessfulCreate	Created pod: isvc-canary-promote-v2-predictor-769cdc7dcc-prp88
19:11:29	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-prp88	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
19:11:29	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-predictor-6b6f66ff94 from 0 to 2
19:11:29	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-6b6f66ff94	SuccessfulCreate	Created pod: isvc-canary-promote-predictor-6b6f66ff94-kthkb
19:11:30	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-cn57m	Created	Created container: storage-initializer
19:11:30	kserve-ci-e2e-test	multus	isvc-canary-promote-predictor-6b6f66ff94-cn57m	AddedInterface	Add eth0 [10.133.0.43/23] from ovn-kubernetes
19:11:30	kserve-ci-e2e-test	multus	isvc-canary-promote-predictor-6b6f66ff94-kthkb	AddedInterface	Add eth0 [10.134.0.48/23] from ovn-kubernetes
19:11:30	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-cn57m	Started	Started container storage-initializer
19:11:30	kserve-ci-e2e-test	multus	isvc-canary-promote-v2-predictor-769cdc7dcc-prp88	AddedInterface	Add eth0 [10.134.0.47/23] from ovn-kubernetes
19:11:30	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-prp88	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:11:30	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-cn57m	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:11:30	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-kthkb	Started	Started container storage-initializer
19:11:30	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-prp88	Created	Created container: storage-initializer
19:11:30	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-prp88	Started	Started container storage-initializer
19:11:30	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-kthkb	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:11:30	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-kthkb	Created	Created container: storage-initializer
19:11:32 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:11:32 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:11:34	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-kthkb	Started	Started container kserve-container
19:11:34	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-kthkb	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" already present on machine
19:11:34	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-kthkb	Started	Started container kube-rbac-proxy
19:11:34	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-kthkb	Created	Created container: kube-rbac-proxy
19:11:34	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-kthkb	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:11:34	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-kthkb	Created	Created container: kserve-container
19:11:35	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-cn57m	Created	Created container: kserve-container
19:11:35	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-cn57m	Started	Started container kserve-container
19:11:35	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-prp88	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" already present on machine
19:11:35	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-prp88	Started	Started container kube-rbac-proxy
19:11:35	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-prp88	Created	Created container: kube-rbac-proxy
19:11:35	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-cn57m	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" already present on machine
19:11:35	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-prp88	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:11:35	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-prp88	Created	Created container: kserve-container
19:11:35	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-cn57m	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:11:35	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-cn57m	Created	Created container: kube-rbac-proxy
19:11:35	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-cn57m	Started	Started container kube-rbac-proxy
19:11:35	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-prp88	Started	Started container kserve-container
19:11:49 (x7)	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-76cc455874-rktzp	Unhealthy	Readiness probe failed: Get "https://10.134.0.45:8643/healthz": dial tcp 10.134.0.45:8643: connect: connection refused
19:12:05 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-866dcdbf79-5gnxs	Unhealthy	Readiness probe failed: dial tcp 10.134.0.46:8080: connect: connection refused
19:12:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	InferenceServiceReady	InferenceService [isvc-sklearn-v2] is Ready
19:12:15 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:12:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-866dcdbf79-5gnxs	Killing	Stopping container kube-rbac-proxy
19:12:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-866dcdbf79-5gnxs	Killing	Stopping container kserve-container
19:12:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:12:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:12:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-866dcdbf79-5gnxs	Unhealthy	Readiness probe failed: Get "https://10.134.0.46:8643/healthz": dial tcp 10.134.0.46:8643: connect: connection refused
19:12:23	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-mixed-predictor-6c8c596f78	SuccessfulCreate	Created pod: isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j
19:12:23	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-mixed": the object has been modified; please apply your changes to the latest version and try again
19:12:23 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-mixed": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-mixed": the object has been modified; please apply your changes to the latest version and try again
19:12:23	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-mixed-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-mixed-predictor-6c8c596f78 from 0 to 1
19:12:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j	Started	Started container storage-initializer
19:12:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:12:24	kserve-ci-e2e-test	multus	isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j	AddedInterface	Add eth0 [10.134.0.49/23] from ovn-kubernetes
19:12:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j	Created	Created container: storage-initializer
19:12:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j	Started	Started container kube-rbac-proxy
19:12:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" already present on machine
19:12:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j	Created	Created container: kserve-container
19:12:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j	Started	Started container kserve-container
19:12:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j	Created	Created container: kube-rbac-proxy
19:12:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:12:43 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-cn57m	Unhealthy	Readiness probe failed: dial tcp 10.133.0.43:8080: connect: connection refused
19:12:48 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-kthkb	Unhealthy	Readiness probe failed: dial tcp 10.134.0.48:8080: connect: connection refused
19:12:51 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-prp88	Unhealthy	Readiness probe failed: dial tcp 10.134.0.47:8080: connect: connection refused
19:12:53 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:12:53 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:13:01	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-kthkb	Killing	Stopping container kube-rbac-proxy
19:13:01 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	UpdateFailed	Failed to update status for InferenceService "isvc-canary-promote": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again
19:13:01	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-kthkb	Killing	Stopping container kserve-container
19:13:01	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-6b6f66ff94	SuccessfulDelete	Deleted pod: isvc-canary-promote-predictor-6b6f66ff94-kthkb
19:13:01	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-promote-predictor-6b6f66ff94 from 2 to 1
19:13:01 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again
19:13:01 (x21)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:13:02	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-kthkb	Unhealthy	Readiness probe failed: Get "https://10.134.0.48:8643/healthz": dial tcp 10.134.0.48:8643: connect: connection refused
19:13:04	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-cn57m	Killing	Stopping container kserve-container
19:13:04	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-cn57m	Killing	Stopping container kube-rbac-proxy
19:13:04	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-v2-predictor-769cdc7dcc from 1 to 2
19:13:05	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:13:05	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj	Started	Started container storage-initializer
19:13:05	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-v2-predictor-769cdc7dcc	SuccessfulCreate	Created pod: isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj
19:13:05	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj	Created	Created container: storage-initializer
19:13:05	kserve-ci-e2e-test	multus	isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj	AddedInterface	Add eth0 [10.133.0.44/23] from ovn-kubernetes
19:13:08	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-6b6f66ff94-cn57m	Unhealthy	Readiness probe failed: Get "https://10.133.0.43:8643/healthz": dial tcp 10.133.0.43:8643: connect: connection refused
19:13:09	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj	Created	Created container: kube-rbac-proxy
19:13:09	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" already present on machine
19:13:09	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj	Created	Created container: kserve-container
19:13:09	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj	Started	Started container kube-rbac-proxy
19:13:09	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj	Started	Started container kserve-container
19:13:09	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:13:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:13:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:13:46	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	InferenceServiceReady	InferenceService [isvc-sklearn-v2-mixed] is Ready
19:13:46 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:13:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j	Killing	Stopping container kube-rbac-proxy
19:13:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j	Killing	Stopping container kserve-container
19:13:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j	Unhealthy	Readiness probe failed: Get "https://10.134.0.49:8643/healthz": dial tcp 10.134.0.49:8643: connect: connection refused
19:13:55 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j	Unhealthy	Readiness probe failed: dial tcp 10.134.0.49:8080: connect: connection refused
19:13:58	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow": the object has been modified; please apply your changes to the latest version and try again
19:13:58	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-xch6v	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-tensorflow-predictor-serving-cert" not found
19:13:58 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow": the object has been modified; please apply your changes to the latest version and try again
19:13:58	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-predictor-547dfc5669 from 0 to 1
19:13:58	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-predictor-547dfc5669	SuccessfulCreate	Created pod: isvc-tensorflow-predictor-547dfc5669-xch6v
19:13:59	kserve-ci-e2e-test	multus	isvc-tensorflow-predictor-547dfc5669-xch6v	AddedInterface	Add eth0 [10.134.0.50/23] from ovn-kubernetes
19:13:59	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-xch6v	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:13:59	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-xch6v	Created	Created container: storage-initializer
19:13:59	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-xch6v	Started	Started container storage-initializer
19:14:05	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-xch6v	Pulling	Pulling image "tensorflow/serving:2.6.2"
19:14:08	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-xch6v	Started	Started container kube-rbac-proxy
19:14:08	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-xch6v	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 3.532s (3.532s including waiting). Image size: 425873876 bytes.
19:14:08	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-xch6v	Created	Created container: kserve-container
19:14:08	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-xch6v	Started	Started container kserve-container
19:14:08	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-xch6v	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:14:08	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-xch6v	Created	Created container: kube-rbac-proxy
19:14:16 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-xch6v	Unhealthy	Readiness probe failed: dial tcp 10.134.0.50:8080: connect: connection refused
19:14:26	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	InferenceServiceReady	InferenceService [isvc-tensorflow] is Ready
19:14:26 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	InferenceServiceReady	InferenceService [isvc-canary-promote] is Ready
19:14:26 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:14:28 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:14:28 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:14:35	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj	Killing	Stopping container kube-rbac-proxy
19:14:35	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-prp88	Killing	Stopping container kube-rbac-proxy
19:14:35	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-prp88	Killing	Stopping container kserve-container
19:14:35	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj	Killing	Stopping container kserve-container
19:14:36	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj	Unhealthy	Readiness probe failed: Get "https://10.133.0.44:8643/healthz": dial tcp 10.133.0.44:8643: connect: connection refused
19:14:36 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-mxlcj	Unhealthy	Readiness probe failed: dial tcp 10.133.0.44:8080: connect: connection refused
19:14:37	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-769cdc7dcc-prp88	Unhealthy	Readiness probe failed: Get "https://10.134.0.47:8643/healthz": dial tcp 10.134.0.47:8643: connect: connection refused
19:14:39	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-xch6v	Killing	Stopping container kube-rbac-proxy
19:14:39	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-xch6v	Killing	Stopping container kserve-container
19:14:40	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-c9f4b9d7d	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg
19:14:40	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-predictor-c9f4b9d7d from 0 to 2
19:14:40	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-v2-predictor-6dc8d44cf9 from 0 to 1
19:14:40	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-c9f4b9d7d	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl
19:14:40	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-predictor-serving-cert" not found
19:14:40	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-v2-predictor-6dc8d44cf9	SuccessfulCreate	Created pod: isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs
19:14:41	kserve-ci-e2e-test	multus	isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs	AddedInterface	Add eth0 [10.134.0.51/23] from ovn-kubernetes
19:14:41	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl	Started	Started container storage-initializer
19:14:41	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:14:41	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs	Started	Started container storage-initializer
19:14:41	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs	Created	Created container: storage-initializer
19:14:41	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl	AddedInterface	Add eth0 [10.133.0.45/23] from ovn-kubernetes
19:14:41	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg	Started	Started container storage-initializer
19:14:41	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg	Created	Created container: storage-initializer
19:14:41	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:14:41	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl	Created	Created container: storage-initializer
19:14:41	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:14:41	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg	AddedInterface	Add eth0 [10.134.0.52/23] from ovn-kubernetes
19:14:44	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
19:14:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow-runtime": the object has been modified; please apply your changes to the latest version and try again
19:14:44	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-runtime-predictor-855cfcd677 from 0 to 1
19:14:44	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-runtime-predictor-855cfcd677	SuccessfulCreate	Created pod: isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt
19:14:45	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs	Started	Started container kserve-container
19:14:45	kserve-ci-e2e-test	multus	isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt	AddedInterface	Add eth0 [10.133.0.46/23] from ovn-kubernetes
19:14:45	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:14:45	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs	Created	Created container: kube-rbac-proxy
19:14:45	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs	Started	Started container kube-rbac-proxy
19:14:45	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs	Created	Created container: kserve-container
19:14:45	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" already present on machine
19:14:45	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:14:45	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:14:45	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg	Started	Started container kube-rbac-proxy
19:14:45	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg	Created	Created container: kube-rbac-proxy
19:14:45	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" already present on machine
19:14:45	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg	Created	Created container: kserve-container
19:14:45	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg	Started	Started container kserve-container
19:14:46	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl	Created	Created container: kube-rbac-proxy
19:14:46	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl	Started	Started container kube-rbac-proxy
19:14:46	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" already present on machine
19:14:46	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl	Created	Created container: kserve-container
19:14:46	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl	Started	Started container kserve-container
19:14:46	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:14:46	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt	Started	Started container storage-initializer
19:14:46	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt	Created	Created container: storage-initializer
19:14:51	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt	Pulling	Pulling image "tensorflow/serving:2.6.2"
19:14:54	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt	Started	Started container kube-rbac-proxy
19:14:54	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt	Created	Created container: kube-rbac-proxy
19:14:54	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt	Started	Started container kserve-container
19:14:54	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 3.631s (3.631s including waiting). Image size: 425873876 bytes.
19:14:54	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt	Created	Created container: kserve-container
19:14:54	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:15:02 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt	Unhealthy	Readiness probe failed: dial tcp 10.133.0.46:8080: connect: connection refused
19:15:06 (x6)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-xch6v	Unhealthy	Readiness probe failed: Get "https://10.134.0.50:8643/healthz": dial tcp 10.134.0.50:8643: connect: connection refused
19:15:12 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:15:12	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	InferenceServiceReady	InferenceService [isvc-tensorflow-runtime] is Ready
19:15:14 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:15:14 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:15:26	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt	Killing	Stopping container kserve-container
19:15:26	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt	Killing	Stopping container kube-rbac-proxy
19:15:31 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	UpdateFailed	Failed to update status for InferenceService "isvc-triton": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-triton": the object has been modified; please apply your changes to the latest version and try again
19:15:31	kserve-ci-e2e-test	replicaset-controller	isvc-triton-predictor-5d7db58676	SuccessfulCreate	Created pod: isvc-triton-predictor-5d7db58676-f8flm
19:15:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-triton": the object has been modified; please apply your changes to the latest version and try again
19:15:31	kserve-ci-e2e-test	deployment-controller	isvc-triton-predictor	ScalingReplicaSet	Scaled up replica set isvc-triton-predictor-5d7db58676 from 0 to 1
19:15:32	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-f8flm	Started	Started container storage-initializer
19:15:32	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-f8flm	Created	Created container: storage-initializer
19:15:32	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-f8flm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:15:32	kserve-ci-e2e-test	multus	isvc-triton-predictor-5d7db58676-f8flm	AddedInterface	Add eth0 [10.134.0.53/23] from ovn-kubernetes
19:15:36	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-f8flm	Pulling	Pulling image "nvcr.io/nvidia/tritonserver:23.05-py3"
19:15:52 (x6)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-kf4pt	Unhealthy	Readiness probe failed: Get "https://10.133.0.46:8643/healthz": dial tcp 10.133.0.46:8643: connect: connection refused
19:16:20	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg	Killing	Stopping container kube-rbac-proxy
19:16:20	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-c9f4b9d7d	SuccessfulDelete	Deleted pod: isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg
19:16:20	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-rollback-predictor-c9f4b9d7d from 2 to 1
19:16:20	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-predictor-c9f4b9d7d from 1 to 2
19:16:20 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again
19:16:20	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg	Killing	Stopping container kserve-container
19:16:20	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-c9f4b9d7d	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9
19:16:20 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	UpdateFailed	Failed to update status for InferenceService "isvc-canary-rollback": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again
19:16:20 (x21)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:16:21	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs	Killing	Stopping container kube-rbac-proxy
19:16:21	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs	Killing	Stopping container kserve-container
19:16:40 (x13)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs	Unhealthy	Readiness probe failed: dial tcp 10.134.0.51:8080: connect: connection refused
19:16:43 (x13)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg	Unhealthy	Readiness probe failed: dial tcp 10.134.0.52:8080: connect: connection refused
19:16:48 (x6)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs	Unhealthy	Readiness probe failed: Get "https://10.134.0.51:8643/healthz": dial tcp 10.134.0.51:8643: connect: connection refused
19:16:48 (x6)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg	Unhealthy	Readiness probe failed: Get "https://10.134.0.52:8643/healthz": dial tcp 10.134.0.52:8643: connect: connection refused
19:16:50	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9	AddedInterface	Add eth0 [10.134.0.54/23] from ovn-kubernetes
19:16:51	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:16:51	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs	Unhealthy	Readiness probe failed: dial tcp 10.134.0.51:8080: i/o timeout
19:17:01	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9	Started	Started container storage-initializer
19:17:01	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9	Created	Created container: storage-initializer
19:17:30	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9	Started	Started container kube-rbac-proxy
19:17:30	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" already present on machine
19:17:30	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9	Created	Created container: kserve-container
19:17:30	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9	Started	Started container kserve-container
19:17:30	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:17:30	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9	Created	Created container: kube-rbac-proxy
19:17:31	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-f8flm	Started	Started container kube-rbac-proxy
19:17:31	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-f8flm	Created	Created container: kube-rbac-proxy
19:17:31	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-f8flm	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:17:31	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-f8flm	Created	Created container: kserve-container
19:17:31	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-f8flm	Started	Started container kserve-container
19:17:31	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-f8flm	Pulled	Successfully pulled image "nvcr.io/nvidia/tritonserver:23.05-py3" in 1m54.422s (1m54.422s including waiting). Image size: 12907074623 bytes.
19:17:37 (x3)	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-f8flm	Unhealthy	Readiness probe failed: dial tcp 10.134.0.53:8080: connect: connection refused
19:17:46 (x9)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:17:46 (x9)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:17:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	InferenceServiceReady	InferenceService [isvc-triton] is Ready
19:17:47 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:17:54	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-f8flm	Killing	Stopping container kube-rbac-proxy
19:17:54	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-f8flm	Killing	Stopping container kserve-container
19:17:59	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-predictor-7566d65bf7 from 0 to 1
19:17:59	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-predictor-7566d65bf7	SuccessfulCreate	Created pod: isvc-xgboost-predictor-7566d65bf7-gwfg9
19:17:59 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost": the object has been modified; please apply your changes to the latest version and try again
19:17:59	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost": the object has been modified; please apply your changes to the latest version and try again
19:18:00	kserve-ci-e2e-test	multus	isvc-xgboost-predictor-7566d65bf7-gwfg9	AddedInterface	Add eth0 [10.134.0.55/23] from ovn-kubernetes
19:18:00	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-gwfg9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:18:00	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-gwfg9	Started	Started container storage-initializer
19:18:00	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-gwfg9	Created	Created container: storage-initializer
19:18:04	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-gwfg9	Pulling	Pulling image "kserve/xgbserver:latest"
19:18:22	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-gwfg9	Started	Started container kube-rbac-proxy
19:18:22	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-gwfg9	Started	Started container kserve-container
19:18:22	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-gwfg9	Created	Created container: kube-rbac-proxy
19:18:22	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-gwfg9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:18:22	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-gwfg9	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 17.68s (17.68s including waiting). Image size: 1332365570 bytes.
19:18:22	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-gwfg9	Created	Created container: kserve-container
19:18:38 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9	Unhealthy	Readiness probe failed: dial tcp 10.134.0.54:8080: connect: connection refused
19:18:48 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	InferenceServiceReady	InferenceService [isvc-canary-rollback] is Ready
19:18:50	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9	Killing	Stopping container kube-rbac-proxy
19:18:50	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9	Killing	Stopping container kserve-container
19:18:50	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl	Killing	Stopping container kserve-container
19:18:50	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl	Killing	Stopping container kube-rbac-proxy
19:18:53	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9	Unhealthy	Readiness probe failed: Get "https://10.134.0.54:8643/healthz": dial tcp 10.134.0.54:8643: connect: connection refused
19:18:53 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl	Unhealthy	Readiness probe failed: dial tcp 10.133.0.45:8080: connect: connection refused
19:18:53	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-c9f4b9d7d-fhxwl	Unhealthy	Readiness probe failed: Get "https://10.133.0.45:8643/healthz": dial tcp 10.133.0.45:8643: connect: connection refused
19:18:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again
19:18:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	UpdateFailed	Failed to update status for InferenceService "isvc-canary-stop": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again
19:18:55	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-stop-v2-predictor-854695c447 from 0 to 1
19:18:55	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-v2-predictor-854695c447	SuccessfulCreate	Created pod: isvc-canary-stop-v2-predictor-854695c447-qpcx4
19:18:55	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-stop-predictor-5ccc77f5f4 from 0 to 2
19:18:55	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-5ccc77f5f4	SuccessfulCreate	Created pod: isvc-canary-stop-predictor-5ccc77f5f4-fbb6q
19:18:56	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-854695c447-qpcx4	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
19:18:56	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-fbb6q	Created	Created container: storage-initializer
19:18:56	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-8l7nw	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:18:56	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-8l7nw	Created	Created container: storage-initializer
19:18:56	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-8l7nw	Started	Started container storage-initializer
19:18:56	kserve-ci-e2e-test	multus	isvc-canary-stop-predictor-5ccc77f5f4-8l7nw	AddedInterface	Add eth0 [10.134.0.57/23] from ovn-kubernetes
19:18:56	kserve-ci-e2e-test	multus	isvc-canary-stop-predictor-5ccc77f5f4-fbb6q	AddedInterface	Add eth0 [10.133.0.47/23] from ovn-kubernetes
19:18:56	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-fbb6q	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:18:56	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-5ccc77f5f4	SuccessfulCreate	Created pod: isvc-canary-stop-predictor-5ccc77f5f4-8l7nw
19:18:56	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-fbb6q	Started	Started container storage-initializer
19:18:57	kserve-ci-e2e-test	multus	isvc-canary-stop-v2-predictor-854695c447-qpcx4	AddedInterface	Add eth0 [10.134.0.56/23] from ovn-kubernetes
19:18:57	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-854695c447-qpcx4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:18:57	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-854695c447-qpcx4	Created	Created container: storage-initializer
19:18:57	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-854695c447-qpcx4	Started	Started container storage-initializer
19:18:59 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:18:59 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:19:00	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-8l7nw	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" already present on machine
19:19:01	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-fbb6q	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" already present on machine
19:19:01	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-8l7nw	Created	Created container: kserve-container
19:19:01	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-854695c447-qpcx4	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" already present on machine
19:19:01	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-fbb6q	Created	Created container: kserve-container
19:19:01	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-8l7nw	Started	Started container kube-rbac-proxy
19:19:01	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-8l7nw	Created	Created container: kube-rbac-proxy
19:19:01	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-8l7nw	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:19:01	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-8l7nw	Started	Started container kserve-container
19:19:01	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-fbb6q	Started	Started container kserve-container
19:19:01	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-fbb6q	Started	Started container kube-rbac-proxy
19:19:01	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-fbb6q	Created	Created container: kube-rbac-proxy
19:19:01	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-fbb6q	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:19:02	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-854695c447-qpcx4	Started	Started container kube-rbac-proxy
19:19:02	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-854695c447-qpcx4	Created	Created container: kube-rbac-proxy
19:19:02	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-854695c447-qpcx4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:19:02	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-854695c447-qpcx4	Started	Started container kserve-container
19:19:02	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-854695c447-qpcx4	Created	Created container: kserve-container
19:19:14	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:19:14	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:19:18 (x8)	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-gwfg9	Unhealthy	Readiness probe failed: dial tcp 10.134.0.55:8080: connect: connection refused
19:19:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	InferenceServiceReady	InferenceService [isvc-xgboost] is Ready
19:19:28 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:19:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-gwfg9	Killing	Stopping container kserve-container
19:19:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-gwfg9	Killing	Stopping container kube-rbac-proxy
19:19:33	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-gwfg9	Unhealthy	Readiness probe failed: Get "https://10.134.0.55:8643/healthz": dial tcp 10.134.0.55:8643: connect: connection refused
19:19:34	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-mlserver-predictor-7b55b84f54 from 0 to 1
19:19:34	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-mlserver-predictor-7b55b84f54	SuccessfulCreate	Created pod: isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9
19:19:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
19:19:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
19:19:35	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:19:35	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9	Created	Created container: storage-initializer
19:19:35	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9	Started	Started container storage-initializer
19:19:35	kserve-ci-e2e-test	multus	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9	AddedInterface	Add eth0 [10.133.0.48/23] from ovn-kubernetes
19:19:40	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9	Started	Started container kube-rbac-proxy
19:19:40	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:19:40	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9	Started	Started container kserve-container
19:19:40	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9	Created	Created container: kserve-container
19:19:40	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:19:40	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9	Created	Created container: kube-rbac-proxy
19:20:04 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:20:04 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:20:07 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-8l7nw	Unhealthy	Readiness probe failed: dial tcp 10.134.0.57:8080: connect: connection refused
19:20:11 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-854695c447-qpcx4	Unhealthy	Readiness probe failed: dial tcp 10.134.0.56:8080: connect: connection refused
19:20:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	InferenceServiceReady	InferenceService [isvc-xgboost-v2-mlserver] is Ready
19:20:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	InferenceServiceReady	InferenceService [isvc-canary-stop] is Ready
19:20:18 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:20:19	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:20:19	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:20:21	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-5ccc77f5f4	SuccessfulDelete	Deleted pod: isvc-canary-stop-predictor-5ccc77f5f4-8l7nw
19:20:21	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-8l7nw	Killing	Stopping container kube-rbac-proxy
19:20:21	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-8l7nw	Killing	Stopping container kserve-container
19:20:21	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-stop-predictor-5ccc77f5f4 from 2 to 1
19:20:22	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-8l7nw	Unhealthy	Readiness probe failed: Get "https://10.134.0.57:8643/healthz": dial tcp 10.134.0.57:8643: connect: connection refused
19:20:24	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9	Killing	Stopping container kube-rbac-proxy
19:20:24	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9	Killing	Stopping container kserve-container
19:20:25	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-854695c447-qpcx4	Killing	Stopping container kserve-container
19:20:25	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-854695c447-qpcx4	Killing	Stopping container kube-rbac-proxy
19:20:26	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-fbb6q	Killing	Stopping container kserve-container
19:20:26	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-fbb6q	Killing	Stopping container kube-rbac-proxy
19:20:26 (x23)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:20:27 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-fbb6q	Unhealthy	Readiness probe failed: dial tcp 10.133.0.47:8080: connect: connection refused
19:20:28	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9	Unhealthy	Readiness probe failed: Get "http://10.133.0.48:8080/v2/models/isvc-xgboost-v2-mlserver/ready": dial tcp 10.133.0.48:8080: connect: connection refused
19:20:28	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-854695c447-qpcx4	Unhealthy	Readiness probe failed: Get "https://10.134.0.56:8643/healthz": dial tcp 10.134.0.56:8643: connect: connection refused
19:20:28	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ghfd9	Unhealthy	Readiness probe failed: Get "https://10.133.0.48:8643/healthz": dial tcp 10.133.0.48:8643: connect: connection refused
19:20:29	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5ccc77f5f4-fbb6q	Unhealthy	Readiness probe failed: Get "https://10.133.0.47:8643/healthz": dial tcp 10.133.0.47:8643: connect: connection refused
19:20:30	kserve-ci-e2e-test	deployment-controller	xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set xgboost-v2-mlserver-predictor-59944bcf4c from 0 to 1
19:20:30	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv	Started	Started container storage-initializer
19:20:30	kserve-ci-e2e-test	multus	xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv	AddedInterface	Add eth0 [10.133.0.49/23] from ovn-kubernetes
19:20:30	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
19:20:30	kserve-ci-e2e-test	replicaset-controller	xgboost-v2-mlserver-predictor-59944bcf4c	SuccessfulCreate	Created pod: xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv
19:20:30	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv	Created	Created container: storage-initializer
19:20:30	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:20:35	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv	Started	Started container kserve-container
19:20:35	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv	Created	Created container: kserve-container
19:20:35	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:20:35	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv	Created	Created container: kube-rbac-proxy
19:20:35	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:20:35	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv	Started	Started container kube-rbac-proxy
19:20:36	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-predictor-54f85ddf88 from 0 to 1
19:20:36	kserve-ci-e2e-test	multus	isvc-lightgbm-predictor-54f85ddf88-djvhk	AddedInterface	Add eth0 [10.134.0.58/23] from ovn-kubernetes
19:20:36	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm": the object has been modified; please apply your changes to the latest version and try again
19:20:36	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-predictor-54f85ddf88	SuccessfulCreate	Created pod: isvc-lightgbm-predictor-54f85ddf88-djvhk
19:20:37	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-djvhk	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:20:37	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-djvhk	Started	Started container storage-initializer
19:20:37	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-djvhk	Created	Created container: storage-initializer
19:20:41	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-djvhk	Pulling	Pulling image "kserve/lgbserver:latest"
19:20:45	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:20:45	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:20:47	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-djvhk	Started	Started container kube-rbac-proxy
19:20:47	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-djvhk	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:20:47	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-djvhk	Created	Created container: kube-rbac-proxy
19:20:47	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-djvhk	Pulled	Successfully pulled image "kserve/lgbserver:latest" in 6.268s (6.268s including waiting). Image size: 632192787 bytes.
19:20:47	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-djvhk	Created	Created container: kserve-container
19:20:47	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-djvhk	Started	Started container kserve-container
19:21:12	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	InferenceServiceReady	InferenceService [xgboost-v2-mlserver] is Ready
19:21:13 (x10)	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:21:15 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:21:15 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:21:20	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv	Killing	Stopping container kube-rbac-proxy
19:21:20	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv	Killing	Stopping container kserve-container
19:21:21 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:21:21 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:21:22	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv	Unhealthy	Readiness probe failed: Get "http://10.133.0.49:8080/v2/models/xgboost-v2-mlserver/ready": dial tcp 10.133.0.49:8080: connect: connection refused
19:21:22	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-nfcvv	Unhealthy	Readiness probe failed: Get "https://10.133.0.49:8643/healthz": dial tcp 10.133.0.49:8643: connect: connection refused
19:21:25	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-runtime-predictor-54dd488c85 from 0 to 1
19:21:25	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-runtime": the object has been modified; please apply your changes to the latest version and try again
19:21:25	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-runtime-predictor-54dd488c85	SuccessfulCreate	Created pod: isvc-xgboost-runtime-predictor-54dd488c85-9gxq2
19:21:26	kserve-ci-e2e-test	multus	isvc-xgboost-runtime-predictor-54dd488c85-9gxq2	AddedInterface	Add eth0 [10.134.0.59/23] from ovn-kubernetes
19:21:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-9gxq2	Started	Started container storage-initializer
19:21:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-9gxq2	Created	Created container: storage-initializer
19:21:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-9gxq2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:21:30	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-9gxq2	Started	Started container kube-rbac-proxy
19:21:30	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-9gxq2	Created	Created container: kube-rbac-proxy
19:21:30	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-9gxq2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:21:30	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-9gxq2	Started	Started container kserve-container
19:21:30	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-9gxq2	Created	Created container: kserve-container
19:21:30	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-9gxq2	Pulled	Container image "kserve/xgbserver:latest" already present on machine
19:21:55 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:21:55 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:21:55 (x9)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-djvhk	Unhealthy	Readiness probe failed: dial tcp 10.134.0.58:8080: connect: connection refused
19:22:05 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:22:05	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	InferenceServiceReady	InferenceService [isvc-lightgbm] is Ready
19:22:06	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-djvhk	Killing	Stopping container kserve-container
19:22:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:22:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:22:06	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-djvhk	Killing	Stopping container kube-rbac-proxy
19:22:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-djvhk	Unhealthy	Readiness probe failed: Get "https://10.134.0.58:8643/healthz": dial tcp 10.134.0.58:8643: connect: connection refused
19:22:11	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-runtime-predictor-577f7db7cb from 0 to 1
19:22:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-runtime": the object has been modified; please apply your changes to the latest version and try again
19:22:12	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8	Started	Started container storage-initializer
19:22:12	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:22:12	kserve-ci-e2e-test	multus	isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8	AddedInterface	Add eth0 [10.134.0.60/23] from ovn-kubernetes
19:22:12	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8	Created	Created container: storage-initializer
19:22:12	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-runtime-predictor-577f7db7cb	SuccessfulCreate	Created pod: isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8
19:22:16	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8	Created	Created container: kserve-container
19:22:16	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8	Pulled	Container image "kserve/lgbserver:latest" already present on machine
19:22:17	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:22:17	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8	Started	Started container kserve-container
19:22:17	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8	Created	Created container: kube-rbac-proxy
19:22:17	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8	Started	Started container kube-rbac-proxy
19:22:37	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-runtime] is Ready
19:22:38 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:22:40 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:22:40 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:22:41 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:22:41 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:22:45	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-9gxq2	Killing	Stopping container kube-rbac-proxy
19:22:45	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-9gxq2	Killing	Stopping container kserve-container
19:22:47 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-9gxq2	Unhealthy	Readiness probe failed: dial tcp 10.134.0.59:8080: connect: connection refused
19:22:47	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-9gxq2	Unhealthy	Readiness probe failed: Get "https://10.134.0.59:8643/healthz": dial tcp 10.134.0.59:8643: connect: connection refused
19:22:50	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-runtime-predictor-7787c55dc9	SuccessfulCreate	Created pod: isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k
19:22:50	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
19:22:50	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-runtime-predictor-7787c55dc9 from 0 to 1
19:22:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
19:22:51	kserve-ci-e2e-test	multus	isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k	AddedInterface	Add eth0 [10.133.0.50/23] from ovn-kubernetes
19:22:51	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:22:51	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k	Created	Created container: storage-initializer
19:22:52	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k	Started	Started container storage-initializer
19:22:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:22:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k	Created	Created container: kserve-container
19:22:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k	Started	Started container kserve-container
19:22:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:22:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k	Created	Created container: kube-rbac-proxy
19:22:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k	Started	Started container kube-rbac-proxy
19:23:20 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:23:20 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:23:33	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
19:23:35 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:23:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-runtime] is Ready
19:23:42	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8	Killing	Stopping container kserve-container
19:23:42	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8	Killing	Stopping container kube-rbac-proxy
19:23:42 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:23:42 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:23:43	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-v2-runtime] is Ready
19:23:43 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:23:44	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8	Unhealthy	Readiness probe failed: Get "https://10.134.0.60:8643/healthz": dial tcp 10.134.0.60:8643: connect: connection refused
19:23:44 (x10)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8	Unhealthy	Readiness probe failed: dial tcp 10.134.0.60:8080: connect: connection refused
19:23:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
19:23:47	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-runtime-predictor-7f4957c5d5 from 0 to 1
19:23:48	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln	Started	Started container storage-initializer
19:23:48	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5	SuccessfulCreate	Created pod: isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln
19:23:48	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:23:48	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln	Created	Created container: storage-initializer
19:23:48	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln	AddedInterface	Add eth0 [10.133.0.51/23] from ovn-kubernetes
19:23:50	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k	Killing	Stopping container kserve-container
19:23:50 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:23:50 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:23:50	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k	Killing	Stopping container kube-rbac-proxy
19:23:52	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:23:52	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:23:52	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln	Created	Created container: kserve-container
19:23:52	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln	Started	Started container kserve-container
19:23:52	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln	Created	Created container: kube-rbac-proxy
19:23:52	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln	Started	Started container kube-rbac-proxy
19:23:53	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-sh27k	Unhealthy	Readiness probe failed: Get "https://10.133.0.50:8643/healthz": dial tcp 10.133.0.50:8643: connect: connection refused
19:23:55	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-predictor-fcc668bc from 0 to 1
19:23:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
19:23:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-w5xtm	Created	Created container: storage-initializer
19:23:56	kserve-ci-e2e-test	multus	isvc-xgboost-v2-predictor-fcc668bc-w5xtm	AddedInterface	Add eth0 [10.134.0.61/23] from ovn-kubernetes
19:23:56	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-predictor-fcc668bc	SuccessfulCreate	Created pod: isvc-xgboost-v2-predictor-fcc668bc-w5xtm
19:23:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-w5xtm	Started	Started container storage-initializer
19:23:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-w5xtm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:24:01	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-w5xtm	Started	Started container kube-rbac-proxy
19:24:01	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-w5xtm	Pulled	Container image "kserve/xgbserver:latest" already present on machine
19:24:01	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-w5xtm	Created	Created container: kube-rbac-proxy
19:24:01	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-w5xtm	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:24:01	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-w5xtm	Started	Started container kserve-container
19:24:01	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-w5xtm	Created	Created container: kserve-container
19:24:25 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:24:25 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:24:29 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:24:29	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-runtime] is Ready
19:24:33 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:24:33 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:24:38	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln	Killing	Stopping container kserve-container
19:24:38	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6fgln	Killing	Stopping container kube-rbac-proxy
19:24:43	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
19:24:43	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5 from 0 to 1
19:24:43	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5	SuccessfulCreate	Created pod: isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r
19:24:43	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r	Started	Started container storage-initializer
19:24:43	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r	Created	Created container: storage-initializer
19:24:43	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:24:43	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r	AddedInterface	Add eth0 [10.134.0.62/23] from ovn-kubernetes
19:24:48	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:24:48	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r	Started	Started container kube-rbac-proxy
19:24:48	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r	Created	Created container: kube-rbac-proxy
19:24:48	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r	Started	Started container kserve-container
19:24:48	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r	Created	Created container: kserve-container
19:24:48	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r	Pulled	Container image "kserve/lgbserver:latest" already present on machine
19:24:51 (x2)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r	Unhealthy	Readiness probe failed: dial tcp 10.134.0.62:8080: connect: connection refused
19:24:56	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-kserve] is Ready
19:24:56 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:24:58	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:24:58	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:25:03	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r	Killing	Stopping container kube-rbac-proxy
19:25:03	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r	Killing	Stopping container kserve-container
19:25:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-mlflow-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-mlflow-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
19:25:08	kserve-ci-e2e-test	deployment-controller	isvc-mlflow-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-mlflow-v2-runtime-predictor-6456f94494 from 0 to 1
19:25:08	kserve-ci-e2e-test	replicaset-controller	isvc-mlflow-v2-runtime-predictor-6456f94494	SuccessfulCreate	Created pod: isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc
19:25:09	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	InferenceServiceReady	InferenceService [isvc-xgboost-v2] is Ready
19:25:09 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:25:09	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc	FailedMount	MountVolume.SetUp failed for volume "kube-api-access-hhj9j" : failed to sync configmap cache: timed out waiting for the condition
19:25:10 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:25:10 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:25:10	kserve-ci-e2e-test	multus	isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc	AddedInterface	Add eth0 [10.133.0.52/23] from ovn-kubernetes
19:25:10	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:25:10	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc	Created	Created container: storage-initializer
19:25:10	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc	Started	Started container storage-initializer
19:25:15	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc	Created	Created container: kserve-container
19:25:15	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:25:15	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc	Started	Started container kserve-container
19:25:15	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:25:15	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc	Created	Created container: kube-rbac-proxy
19:25:15	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc	Started	Started container kube-rbac-proxy
19:25:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-w5xtm	Killing	Stopping container kube-rbac-proxy
19:25:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-w5xtm	Killing	Stopping container kserve-container
19:25:19	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-w5xtm	Unhealthy	Readiness probe failed: Get "https://10.134.0.61:8643/healthz": dial tcp 10.134.0.61:8643: connect: connection refused
19:25:19 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-w5xtm	Unhealthy	Readiness probe failed: dial tcp 10.134.0.61:8080: connect: connection refused
19:25:21	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-predictor-6786d494d6	SuccessfulCreate	Created pod: isvc-sklearn-s3-predictor-6786d494d6-9vddp
19:25:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6786d494d6-9vddp	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-predictor-serving-cert" not found
19:25:21 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3": the object has been modified; please apply your changes to the latest version and try again
19:25:21	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-predictor-6786d494d6 from 0 to 1
19:25:21	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3": the object has been modified; please apply your changes to the latest version and try again
19:25:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6786d494d6-9vddp	Created	Created container: storage-initializer
19:25:22	kserve-ci-e2e-test	multus	isvc-sklearn-s3-predictor-6786d494d6-9vddp	AddedInterface	Add eth0 [10.134.0.63/23] from ovn-kubernetes
19:25:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6786d494d6-9vddp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:25:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6786d494d6-9vddp	Started	Started container storage-initializer
19:25:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6786d494d6-9vddp	Started	Started container kube-rbac-proxy
19:25:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6786d494d6-9vddp	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" already present on machine
19:25:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6786d494d6-9vddp	Created	Created container: kserve-container
19:25:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6786d494d6-9vddp	Created	Created container: kube-rbac-proxy
19:25:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6786d494d6-9vddp	Started	Started container kserve-container
19:25:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6786d494d6-9vddp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:25:38 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:25:38 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:25:51	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:25:51	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:25:52 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:25:52	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	InferenceServiceReady	InferenceService [isvc-mlflow-v2-runtime] is Ready
19:25:53	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:25:53	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:25:58	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc	Killing	Stopping container kube-rbac-proxy
19:25:58	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-4trgc	Killing	Stopping container kserve-container
19:26:03	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-mcp-predictor-76d4c69869	SuccessfulCreate	Created pod: isvc-sklearn-mcp-predictor-76d4c69869-4jtdv
19:26:03 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-mcp": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-mcp": the object has been modified; please apply your changes to the latest version and try again
19:26:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-mcp": the object has been modified; please apply your changes to the latest version and try again
19:26:03	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-mcp-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-mcp-predictor-76d4c69869 from 0 to 1
19:26:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-76d4c69869-4jtdv	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
19:26:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-76d4c69869-4jtdv	Created	Created container: storage-initializer
19:26:05	kserve-ci-e2e-test	multus	isvc-sklearn-mcp-predictor-76d4c69869-4jtdv	AddedInterface	Add eth0 [10.134.0.64/23] from ovn-kubernetes
19:26:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-76d4c69869-4jtdv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:26:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-76d4c69869-4jtdv	Started	Started container storage-initializer
19:26:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-76d4c69869-4jtdv	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" already present on machine
19:26:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-76d4c69869-4jtdv	Pulling	Pulling image "quay.io/opendatahub/kserve-agent:latest"
19:26:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-76d4c69869-4jtdv	Created	Created container: kserve-container
19:26:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-76d4c69869-4jtdv	Started	Started container kserve-container
19:26:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-76d4c69869-4jtdv	Created	Created container: kserve-agent
19:26:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-76d4c69869-4jtdv	Started	Started container kserve-agent
19:26:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-76d4c69869-4jtdv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:26:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-76d4c69869-4jtdv	Created	Created container: kube-rbac-proxy
19:26:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-76d4c69869-4jtdv	Started	Started container kube-rbac-proxy
19:26:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-76d4c69869-4jtdv	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent:latest" in 2.592s (2.592s including waiting). Image size: 239481897 bytes.
19:26:30 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6786d494d6-9vddp	Unhealthy	Readiness probe failed: dial tcp 10.134.0.63:8080: connect: connection refused
19:26:36 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:26:36 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:26:41	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858 from 0 to 1
19:26:41	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4
19:26:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	InferenceServiceReady	InferenceService [isvc-sklearn-s3] is Ready
19:26:41 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-pass": the object has been modified; please apply your changes to the latest version and try again
19:26:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-pass": the object has been modified; please apply your changes to the latest version and try again
19:26:41 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:26:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6786d494d6-9vddp	Killing	Stopping container kserve-container
19:26:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6786d494d6-9vddp	Killing	Stopping container kube-rbac-proxy
19:26:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4	Started	Started container storage-initializer
19:26:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4	Created	Created container: storage-initializer
19:26:42	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4	AddedInterface	Add eth0 [10.133.0.53/23] from ovn-kubernetes
19:26:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:26:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4	Started	Started container kserve-container
19:26:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" already present on machine
19:26:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4	Created	Created container: kserve-container
19:26:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:26:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4	Created	Created container: kube-rbac-proxy
19:26:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4	Started	Started container kube-rbac-proxy
19:26:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-76d4c69869-4jtdv	Unhealthy	Liveness probe failed: Get "http://10.134.0.64:8080/v1/models/isvc-sklearn-mcp": dial tcp 10.134.0.64:8080: connect: connection refused
19:26:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6786d494d6-9vddp	Unhealthy	Readiness probe failed: Get "https://10.134.0.63:8643/healthz": dial tcp 10.134.0.63:8643: connect: connection refused
19:27:11 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:27:11 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:27:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	InferenceServiceReady	InferenceService [isvc-sklearn-mcp] is Ready
19:27:19 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:27:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-76d4c69869-4jtdv	Killing	Stopping container kserve-container
19:27:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-76d4c69869-4jtdv	Killing	Stopping container kube-rbac-proxy
19:27:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-76d4c69869-4jtdv	Killing	Stopping container kserve-agent
19:27:29	kserve-ci-e2e-test	deployment-controller	isvc-paddle-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-predictor-6d5c9fd4f6 from 0 to 1
19:27:29	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-predictor-6d5c9fd4f6	SuccessfulCreate	Created pod: isvc-paddle-predictor-6d5c9fd4f6-4fvsm
19:27:29	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4fvsm	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-paddle-predictor-serving-cert" not found
19:27:29	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	UpdateFailed	Failed to update status for InferenceService "isvc-paddle": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle": the object has been modified; please apply your changes to the latest version and try again
19:27:30	kserve-ci-e2e-test	multus	isvc-paddle-predictor-6d5c9fd4f6-4fvsm	AddedInterface	Add eth0 [10.133.0.54/23] from ovn-kubernetes
19:27:30	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4fvsm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:27:30	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4fvsm	Created	Created container: storage-initializer
19:27:30	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4fvsm	Started	Started container storage-initializer
19:27:35	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4fvsm	Pulling	Pulling image "kserve/paddleserver:latest"
19:27:47	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4fvsm	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:27:47	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4fvsm	Created	Created container: kube-rbac-proxy
19:27:47	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4fvsm	Started	Started container kserve-container
19:27:47	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4fvsm	Created	Created container: kserve-container
19:27:47	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4fvsm	Pulled	Successfully pulled image "kserve/paddleserver:latest" in 11.984s (11.984s including waiting). Image size: 1175702273 bytes.
19:27:47	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4fvsm	Started	Started container kube-rbac-proxy
19:27:49 (x6)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-76d4c69869-4jtdv	Unhealthy	Readiness probe failed: Get "https://10.134.0.64:8643/healthz": dial tcp 10.134.0.64:8643: connect: connection refused
19:27:49 (x4)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-76d4c69869-4jtdv	Unhealthy	Readiness probe failed: Get "http://10.134.0.64:8080/v1/models/isvc-sklearn-mcp": dial tcp 10.134.0.64:8080: connect: connection refused
19:27:50 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4	Unhealthy	Readiness probe failed: dial tcp 10.133.0.53:8080: connect: connection refused
19:27:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-76d4c69869-4jtdv	Unhealthy	Readiness probe failed: Get "https://10.134.0.64:8643/healthz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
19:27:56 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:27:56 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:27:59 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:27:59 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:28:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-global-pass] is Ready
19:28:00 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:28:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4	Killing	Stopping container kube-rbac-proxy
19:28:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4	Killing	Stopping container kserve-container
19:28:02	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br
19:28:02	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d from 0 to 1
19:28:02	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-fail": the object has been modified; please apply your changes to the latest version and try again
19:28:03	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br	AddedInterface	Add eth0 [10.134.0.65/23] from ovn-kubernetes
19:28:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6f675c8858-bcnh4	Unhealthy	Readiness probe failed: Get "https://10.133.0.53:8643/healthz": dial tcp 10.133.0.53:8643: connect: connection refused
19:28:07 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br	Created	Created container: storage-initializer
19:28:07 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br	Started	Started container storage-initializer
19:28:07 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:28:10	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br	BackOff	Back-off restarting failed container storage-initializer in pod isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br_kserve-ci-e2e-test(9011415f-d141-449b-b542-10629dfd62ba)
19:28:10 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:28:13	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d from 0 to 1
19:28:13	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c
19:28:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-pass": the object has been modified; please apply your changes to the latest version and try again
19:28:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
19:28:14 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-pass": the object has been modified; please apply your changes to the latest version and try again
19:28:15	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c	AddedInterface	Add eth0 [10.134.0.66/23] from ovn-kubernetes
19:28:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c	Started	Started container storage-initializer
19:28:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c	Created	Created container: storage-initializer
19:28:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:28:16	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" already present on machine
19:28:16	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c	Started	Started container kserve-container
19:28:16	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:28:16	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c	Created	Created container: kube-rbac-proxy
19:28:16	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c	Created	Created container: kserve-container
19:28:16	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c	Started	Started container kube-rbac-proxy
19:28:24 (x6)	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4fvsm	Unhealthy	Readiness probe failed: dial tcp 10.133.0.54:8080: connect: connection refused
19:28:29 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:28:29 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:28:34 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:28:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	InferenceServiceReady	InferenceService [isvc-paddle] is Ready
19:28:40	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4fvsm	Killing	Stopping container kube-rbac-proxy
19:28:40	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-4fvsm	Killing	Stopping container kserve-container
19:28:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:28:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:28:45	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-runtime": the object has been modified; please apply your changes to the latest version and try again
19:28:45	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-m4465	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-paddle-runtime-predictor-serving-cert" not found
19:28:45	kserve-ci-e2e-test	deployment-controller	isvc-paddle-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-runtime-predictor-58b99fff76 from 0 to 1
19:28:45	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-runtime-predictor-58b99fff76	SuccessfulCreate	Created pod: isvc-paddle-runtime-predictor-58b99fff76-m4465
19:28:46	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-m4465	Created	Created container: storage-initializer
19:28:46	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-m4465	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:28:46	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-m4465	Started	Started container storage-initializer
19:28:46	kserve-ci-e2e-test	multus	isvc-paddle-runtime-predictor-58b99fff76-m4465	AddedInterface	Add eth0 [10.134.0.67/23] from ovn-kubernetes
19:28:51	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-m4465	Pulling	Pulling image "kserve/paddleserver:latest"
19:29:03	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-m4465	Started	Started container kube-rbac-proxy
19:29:03	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-m4465	Created	Created container: kserve-container
19:29:03	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-m4465	Pulled	Successfully pulled image "kserve/paddleserver:latest" in 11.483s (11.483s including waiting). Image size: 1175702273 bytes.
19:29:03	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-m4465	Created	Created container: kube-rbac-proxy
19:29:03	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-m4465	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:29:03	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-m4465	Started	Started container kserve-container
19:29:29 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:29:29 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:29:34 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:29:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-custom-pass] is Ready
19:29:40 (x6)	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-m4465	Unhealthy	Readiness probe failed: dial tcp 10.134.0.67:8080: connect: connection refused
19:29:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c	Killing	Stopping container kube-rbac-proxy
19:29:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c	Killing	Stopping container kserve-container
19:29:44 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c	Unhealthy	Readiness probe failed: dial tcp 10.134.0.66:8080: connect: connection refused
19:29:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c	Unhealthy	Readiness probe failed: Get "https://10.134.0.66:8643/healthz": dial tcp 10.134.0.66:8643: connect: connection refused
19:29:45 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:29:45	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-fail": the object has been modified; please apply your changes to the latest version and try again
19:29:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
19:29:45	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d from 0 to 1
19:29:45	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh
19:29:45 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:29:46	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh	AddedInterface	Add eth0 [10.133.0.55/23] from ovn-kubernetes
19:29:51 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh	Started	Started container storage-initializer
19:29:51 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:29:51 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:29:51 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:29:51 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh	Created	Created container: storage-initializer
19:29:51	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	InferenceServiceReady	InferenceService [isvc-paddle-runtime] is Ready
19:29:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-676b955b5d-d26fh	Killing	Stopping container storage-initializer
19:29:56	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-pass": the object has been modified; please apply your changes to the latest version and try again
19:29:56	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t	AddedInterface	Add eth0 [10.134.0.68/23] from ovn-kubernetes
19:29:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t	Created	Created container: storage-initializer
19:29:56	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-m4465	Killing	Stopping container kube-rbac-proxy
19:29:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:29:56	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-m4465	Killing	Stopping container kserve-container
19:29:56	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t
19:29:56	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf from 0 to 1
19:29:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t	Started	Started container storage-initializer
19:29:56 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-pass": the object has been modified; please apply your changes to the latest version and try again
19:29:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t	Created	Created container: kserve-container
19:29:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:29:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t	Started	Started container kube-rbac-proxy
19:29:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t	Created	Created container: kube-rbac-proxy
19:29:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" already present on machine
19:29:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t	Started	Started container kserve-container
19:30:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
19:30:01	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-v2-kserve-predictor-64d4c74b4d	SuccessfulCreate	Created pod: isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps
19:30:01	kserve-ci-e2e-test	deployment-controller	isvc-paddle-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-v2-kserve-predictor-64d4c74b4d from 0 to 1
19:30:02	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
19:30:03	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps	Started	Started container storage-initializer
19:30:03	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps	Created	Created container: storage-initializer
19:30:03	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:30:03	kserve-ci-e2e-test	multus	isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps	AddedInterface	Add eth0 [10.134.0.69/23] from ovn-kubernetes
19:30:09	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps	Created	Created container: kube-rbac-proxy
19:30:09	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps	Pulled	Container image "kserve/paddleserver:latest" already present on machine
19:30:09	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps	Created	Created container: kserve-container
19:30:09	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps	Started	Started container kserve-container
19:30:09	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:30:09	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps	Started	Started container kube-rbac-proxy
19:30:11	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:30:11	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:30:46 (x6)	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps	Unhealthy	Readiness probe failed: dial tcp 10.134.0.69:8080: connect: connection refused
19:30:46 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:30:46 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:30:56 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:30:56	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	InferenceServiceReady	InferenceService [isvc-paddle-v2-kserve] is Ready
19:31:01	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:31:01	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:31:03	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps	Killing	Stopping container kserve-container
19:31:03	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps	Killing	Stopping container kube-rbac-proxy
19:31:06 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t	Unhealthy	Readiness probe failed: dial tcp 10.134.0.68:8080: connect: connection refused
19:31:08	kserve-ci-e2e-test	deployment-controller	isvc-pmml-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-predictor-8674795f59 from 0 to 1
19:31:08	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-predictor-8674795f59	SuccessfulCreate	Created pod: isvc-pmml-predictor-8674795f59-lmbkk
19:31:08	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-lmbkk	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-pmml-predictor-serving-cert" not found
19:31:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	UpdateFailed	Failed to update status for InferenceService "isvc-pmml": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml": the object has been modified; please apply your changes to the latest version and try again
19:31:09	kserve-ci-e2e-test	multus	isvc-pmml-predictor-8674795f59-lmbkk	AddedInterface	Add eth0 [10.134.0.70/23] from ovn-kubernetes
19:31:09	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-lmbkk	Started	Started container storage-initializer
19:31:09	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-lmbkk	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:31:09	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-lmbkk	Created	Created container: storage-initializer
19:31:11 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:31:11 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:31:14	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-lmbkk	Pulling	Pulling image "kserve/pmmlserver:latest"
19:31:16 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:31:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-serving-pass] is Ready
19:31:21	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-lmbkk	Started	Started container kube-rbac-proxy
19:31:21	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-lmbkk	Pulled	Successfully pulled image "kserve/pmmlserver:latest" in 7.217s (7.217s including waiting). Image size: 826884477 bytes.
19:31:21	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-lmbkk	Created	Created container: kserve-container
19:31:21	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-lmbkk	Started	Started container kserve-container
19:31:21	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-lmbkk	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:31:21	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-lmbkk	Created	Created container: kube-rbac-proxy
19:31:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t	Killing	Stopping container kube-rbac-proxy
19:31:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t	Killing	Stopping container kserve-container
19:31:27 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-fail": the object has been modified; please apply your changes to the latest version and try again
19:31:27	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9 from 0 to 1
19:31:27	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22
19:31:27	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-fail": the object has been modified; please apply your changes to the latest version and try again
19:31:28	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22	AddedInterface	Add eth0 [10.134.0.71/23] from ovn-kubernetes
19:31:31 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22	Started	Started container storage-initializer
19:31:31 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:31:31 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22	Created	Created container: storage-initializer
19:31:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t	Unhealthy	Readiness probe failed: Get "https://10.134.0.68:8643/healthz": dial tcp 10.134.0.68:8643: connect: connection refused
19:31:35 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:31:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22	BackOff	Back-off restarting failed container storage-initializer in pod isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22_kserve-ci-e2e-test(dcb53242-1dd8-4603-8836-07a8c0f99c83)
19:31:38	kserve-ci-e2e-test	deployment-controller	isvc-pmml-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-runtime-predictor-885fc8777 from 0 to 1
19:31:38	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-scj7n	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-pmml-runtime-predictor-serving-cert" not found
19:31:38	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-runtime-predictor-885fc8777	SuccessfulCreate	Created pod: isvc-pmml-runtime-predictor-885fc8777-scj7n
19:31:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-runtime": the object has been modified; please apply your changes to the latest version and try again
19:31:39	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-scj7n	Created	Created container: storage-initializer
19:31:39	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-scj7n	Started	Started container storage-initializer
19:31:39	kserve-ci-e2e-test	multus	isvc-pmml-runtime-predictor-885fc8777-scj7n	AddedInterface	Add eth0 [10.134.0.72/23] from ovn-kubernetes
19:31:39	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-scj7n	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:31:44	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-scj7n	Started	Started container kube-rbac-proxy
19:31:44	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-scj7n	Created	Created container: kserve-container
19:31:44	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-scj7n	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
19:31:44	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-scj7n	Created	Created container: kube-rbac-proxy
19:31:44	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-scj7n	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:31:44	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-scj7n	Started	Started container kserve-container
19:31:53 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:31:53 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:32:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:32:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:32:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:32:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:32:39 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-lmbkk	Unhealthy	Readiness probe failed: dial tcp 10.134.0.70:8080: connect: connection refused
19:32:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	InferenceServiceReady	InferenceService [isvc-pmml] is Ready
19:32:44 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:32:49	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-lmbkk	Killing	Stopping container kube-rbac-proxy
19:32:49	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-lmbkk	Unhealthy	Readiness probe failed: Get "https://10.134.0.70:8643/healthz": dial tcp 10.134.0.70:8643: connect: connection refused
19:32:49	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-lmbkk	Killing	Stopping container kserve-container
19:32:54	kserve-ci-e2e-test	deployment-controller	isvc-primary-63d815-predictor	ScalingReplicaSet	Scaled up replica set isvc-primary-63d815-predictor-54888c955d from 0 to 1
19:32:54	kserve-ci-e2e-test	replicaset-controller	isvc-primary-63d815-predictor-54888c955d	SuccessfulCreate	Created pod: isvc-primary-63d815-predictor-54888c955d-rw7wt
19:32:54	kserve-ci-e2e-test	kubelet	isvc-primary-63d815-predictor-54888c955d-rw7wt	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-primary-63d815-predictor-serving-cert" not found
19:32:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-63d815	UpdateFailed	Failed to update status for InferenceService "isvc-primary-63d815": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-63d815": the object has been modified; please apply your changes to the latest version and try again
19:32:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-63d815	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-63d815": the object has been modified; please apply your changes to the latest version and try again
19:32:56	kserve-ci-e2e-test	kubelet	isvc-primary-63d815-predictor-54888c955d-rw7wt	Started	Started container storage-initializer
19:32:56	kserve-ci-e2e-test	multus	isvc-primary-63d815-predictor-54888c955d-rw7wt	AddedInterface	Add eth0 [10.134.0.73/23] from ovn-kubernetes
19:32:56	kserve-ci-e2e-test	kubelet	isvc-primary-63d815-predictor-54888c955d-rw7wt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:32:56	kserve-ci-e2e-test	kubelet	isvc-primary-63d815-predictor-54888c955d-rw7wt	Created	Created container: storage-initializer
19:33:00	kserve-ci-e2e-test	kubelet	isvc-primary-63d815-predictor-54888c955d-rw7wt	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1987" already present on machine
19:33:00	kserve-ci-e2e-test	kubelet	isvc-primary-63d815-predictor-54888c955d-rw7wt	Created	Created container: kserve-container
19:33:00	kserve-ci-e2e-test	kubelet	isvc-primary-63d815-predictor-54888c955d-rw7wt	Started	Started container kserve-container
19:33:00	kserve-ci-e2e-test	kubelet	isvc-primary-63d815-predictor-54888c955d-rw7wt	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:33:00	kserve-ci-e2e-test	kubelet	isvc-primary-63d815-predictor-54888c955d-rw7wt	Created	Created container: kube-rbac-proxy
19:33:00	kserve-ci-e2e-test	kubelet	isvc-primary-63d815-predictor-54888c955d-rw7wt	Started	Started container kube-rbac-proxy
19:33:04 (x11)	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-scj7n	Unhealthy	Readiness probe failed: dial tcp 10.134.0.72:8080: connect: connection refused
19:33:08 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:33:08 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:33:09	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-63d815-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:33:09	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-63d815-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:33:14 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:33:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	InferenceServiceReady	InferenceService [isvc-pmml-runtime] is Ready
19:33:19	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-scj7n	Killing	Stopping container kserve-container
19:33:19	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-scj7n	Killing	Stopping container kube-rbac-proxy
19:33:21	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-scj7n	Unhealthy	Readiness probe failed: Get "https://10.134.0.72:8643/healthz": dial tcp 10.134.0.72:8643: connect: connection refused
19:33:24	kserve-ci-e2e-test	deployment-controller	isvc-pmml-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-v2-kserve-predictor-84cb55486c from 0 to 1
19:33:24	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
19:33:24	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
19:33:25	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt	Started	Started container storage-initializer
19:33:25	kserve-ci-e2e-test	multus	isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt	AddedInterface	Add eth0 [10.134.0.74/23] from ovn-kubernetes
19:33:25	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:33:25	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt	Created	Created container: storage-initializer
19:33:25	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-v2-kserve-predictor-84cb55486c	SuccessfulCreate	Created pod: isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt
19:33:30	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt	Started	Started container kube-rbac-proxy
19:33:30	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt	Created	Created container: kube-rbac-proxy
19:33:30	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:33:30	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt	Started	Started container kserve-container
19:33:30	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt	Created	Created container: kserve-container
19:33:30	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
19:33:55 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:33:55 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:34:07 (x9)	kserve-ci-e2e-test	kubelet	isvc-primary-63d815-predictor-54888c955d-rw7wt	Unhealthy	Readiness probe failed: dial tcp 10.134.0.73:8080: connect: connection refused
19:34:17	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-63d815	InferenceServiceReady	InferenceService [isvc-primary-63d815] is Ready
19:34:17 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-63d815	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:34:39 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-63d815-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:34:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-63d815	UpdateFailed	Failed to update status for InferenceService "isvc-secondary-63d815": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-63d815": the object has been modified; please apply your changes to the latest version and try again
19:34:39	kserve-ci-e2e-test	deployment-controller	isvc-secondary-63d815-predictor	ScalingReplicaSet	Scaled up replica set isvc-secondary-63d815-predictor-56c45dbb9c from 0 to 1
19:34:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-63d815	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-63d815": the object has been modified; please apply your changes to the latest version and try again
19:34:39 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-63d815-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:34:40	kserve-ci-e2e-test	replicaset-controller	isvc-secondary-63d815-predictor-56c45dbb9c	SuccessfulCreate	Created pod: isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f
19:34:40	kserve-ci-e2e-test	kubelet	isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-secondary-63d815-predictor-serving-cert" not found
19:34:41	kserve-ci-e2e-test	multus	isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f	AddedInterface	Add eth0 [10.133.0.56/23] from ovn-kubernetes
19:34:46 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f	Started	Started container storage-initializer
19:34:46 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f	Created	Created container: storage-initializer
19:34:46 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d22994b609214bce65bb4b930496142821ad1e113aba3930763bfdc84da5f0ea" already present on machine
19:34:48 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt	Unhealthy	Readiness probe failed: dial tcp 10.134.0.74:8080: connect: connection refused
19:34:51	kserve-ci-e2e-test	kubelet	isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f	BackOff	Back-off restarting failed container storage-initializer in pod isvc-secondary-63d815-predictor-56c45dbb9c-wfb6f_kserve-ci-e2e-test(21f59e76-7469-442f-9db1-690c88bcf02d)
19:34:51 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-63d815	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:34:52 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:34:52	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	InferenceServiceReady	InferenceService [isvc-pmml-v2-kserve] is Ready
19:34:54	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-63d815-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:34:54	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-63d815-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:34:55 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:34:55 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:34:56	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt	Killing	Stopping container kserve-container
19:34:56	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt	Killing	Stopping container kube-rbac-proxy
19:34:58	kserve-ci-e2e-test	kubelet	isvc-primary-63d815-predictor-54888c955d-rw7wt	Killing	Stopping container kserve-container
19:34:58	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt	Unhealthy	Readiness probe failed: Get "https://10.134.0.74:8643/healthz": dial tcp 10.134.0.74:8643: connect: connection refused
19:34:58	kserve-ci-e2e-test	kubelet	isvc-primary-63d815-predictor-54888c955d-rw7wt	Killing	Stopping container kube-rbac-proxy
19:35:02	kserve-ci-e2e-test	kubelet	isvc-primary-63d815-predictor-54888c955d-rw7wt	Unhealthy	Readiness probe failed: Get "https://10.134.0.73:8643/healthz": dial tcp 10.134.0.73:8643: connect: connection refused