Events

Time	Namespace	Component	RelatedObject	Reason	Message
19:16:35	kserve-ci-e2e-test		isvc-xgboost-predictor-7566d65bf7-km76b	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b to ip-10-0-136-81.ec2.internal
18:56:30	kserve-ci-e2e-test		isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf to ip-10-0-139-72.ec2.internal
19:08:50	kserve-ci-e2e-test		isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8 to ip-10-0-139-72.ec2.internal
19:30:32	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch to ip-10-0-139-72.ec2.internal
19:25:37	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6 to ip-10-0-139-72.ec2.internal
19:27:08	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k to ip-10-0-136-81.ec2.internal
19:27:19	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6 to ip-10-0-136-81.ec2.internal
19:28:50	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v to ip-10-0-136-81.ec2.internal
19:24:16	kserve-ci-e2e-test		isvc-sklearn-s3-predictor-5584f7c89b-l4jlw	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5584f7c89b-l4jlw to ip-10-0-139-72.ec2.internal
18:59:50	kserve-ci-e2e-test		isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565 to ip-10-0-139-72.ec2.internal
19:17:51	kserve-ci-e2e-test		isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd to ip-10-0-136-81.ec2.internal
19:32:00	kserve-ci-e2e-test		isvc-primary-a291d0-predictor-7f984c747b-z922b	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b to ip-10-0-136-81.ec2.internal
19:17:51	kserve-ci-e2e-test		isvc-canary-stop-predictor-69f76fb68d-zjjqs	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-zjjqs to ip-10-0-139-72.ec2.internal
18:52:59	kserve-ci-e2e-test		isvc-predictive-xgboost-predictor-695f978978-cp8h8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8 to ip-10-0-136-81.ec2.internal
19:33:35	kserve-ci-e2e-test		isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp to ip-10-0-136-81.ec2.internal
19:17:51	kserve-ci-e2e-test		isvc-canary-stop-predictor-69f76fb68d-4dh7t	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t to ip-10-0-136-81.ec2.internal
19:07:01	kserve-ci-e2e-test		isvc-sklearn-runtime-predictor-b78d6df9c-vmwll	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll to ip-10-0-136-81.ec2.internal
18:52:24	kserve-ci-e2e-test		isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8 to ip-10-0-136-81.ec2.internal
19:14:46	kserve-ci-e2e-test		isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b to ip-10-0-139-72.ec2.internal
18:50:39	kserve-ci-e2e-test		isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v to ip-10-0-136-81.ec2.internal
19:21:07	kserve-ci-e2e-test		isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k to ip-10-0-139-72.ec2.internal
19:19:15	kserve-ci-e2e-test		xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv	Scheduled	Successfully assigned kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv to ip-10-0-139-72.ec2.internal
19:06:01	kserve-ci-e2e-test		isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp to ip-10-0-139-72.ec2.internal
19:09:35	kserve-ci-e2e-test		isvc-canary-create-predictor-68cd88b8-vwczt	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-vwczt to ip-10-0-139-72.ec2.internal
19:12:34	kserve-ci-e2e-test		isvc-tensorflow-predictor-547dfc5669-stxg2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2 to ip-10-0-136-81.ec2.internal
19:01:38	kserve-ci-e2e-test		isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8 to ip-10-0-136-81.ec2.internal
19:29:07	kserve-ci-e2e-test		isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv to ip-10-0-139-72.ec2.internal
19:13:19	kserve-ci-e2e-test		isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s to ip-10-0-139-72.ec2.internal
19:14:46	kserve-ci-e2e-test		isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t to ip-10-0-136-81.ec2.internal
19:09:35	kserve-ci-e2e-test		isvc-canary-create-predictor-68cd88b8-zvdx9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9 to ip-10-0-136-81.ec2.internal
18:50:54	kserve-ci-e2e-test		isvc-predictive-sklearn-predictor-5f849d69df-zk96k	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-zk96k to ip-10-0-139-72.ec2.internal
19:28:01	kserve-ci-e2e-test		isvc-paddle-runtime-predictor-58b99fff76-ffwrv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ffwrv to ip-10-0-139-72.ec2.internal
19:14:07	kserve-ci-e2e-test		isvc-triton-predictor-5d7db58676-xl8qh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh to ip-10-0-136-81.ec2.internal
19:09:35	kserve-ci-e2e-test		isvc-canary-create-v2-predictor-6c79c974bf-bdhcj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj to ip-10-0-136-81.ec2.internal
18:59:16	kserve-ci-e2e-test		isvc-autogluon-v2-predictor-9684b4cd9-wps2s	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s to ip-10-0-136-81.ec2.internal
19:23:49	kserve-ci-e2e-test		isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8 to ip-10-0-139-72.ec2.internal
19:26:44	kserve-ci-e2e-test		isvc-paddle-predictor-6d5c9fd4f6-hmhx6	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-hmhx6 to ip-10-0-139-72.ec2.internal
19:29:01	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42 to ip-10-0-136-81.ec2.internal
18:54:54	kserve-ci-e2e-test		isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx to ip-10-0-136-81.ec2.internal
19:14:46	kserve-ci-e2e-test		isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt to ip-10-0-139-72.ec2.internal
19:22:53	kserve-ci-e2e-test		isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6 to ip-10-0-139-72.ec2.internal
19:10:49	kserve-ci-e2e-test		isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj to ip-10-0-139-72.ec2.internal
19:09:13	kserve-ci-e2e-test		isvc-sklearn-v2-predictor-5fd47fbf89-w5j95	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95 to ip-10-0-136-81.ec2.internal
19:32:30	kserve-ci-e2e-test		isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8 to ip-10-0-139-72.ec2.internal
19:07:38	kserve-ci-e2e-test		isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h to ip-10-0-139-72.ec2.internal
19:30:44	kserve-ci-e2e-test		isvc-pmml-runtime-predictor-885fc8777-fn2gk	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-fn2gk to ip-10-0-139-72.ec2.internal
19:03:36	kserve-ci-e2e-test		sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl	Scheduled	Successfully assigned kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl to ip-10-0-139-72.ec2.internal
18:54:09	kserve-ci-e2e-test		message-dumper-predictor-6575bcfdfd-c6scw	Scheduled	Successfully assigned kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-c6scw to ip-10-0-139-72.ec2.internal
19:30:14	kserve-ci-e2e-test		isvc-pmml-predictor-8674795f59-6rsz5	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6rsz5 to ip-10-0-139-72.ec2.internal
18:56:04	kserve-ci-e2e-test		isvc-autogluon-v1-predictor-574b5b7495-r5d6n	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n to ip-10-0-136-81.ec2.internal
19:11:20	kserve-ci-e2e-test		isvc-canary-promote-predictor-674955ff8d-njhv9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-njhv9 to ip-10-0-139-72.ec2.internal
19:24:24	kserve-ci-e2e-test		isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg to ip-10-0-139-72.ec2.internal
19:11:20	kserve-ci-e2e-test		isvc-canary-promote-predictor-674955ff8d-ns5mk	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk to ip-10-0-136-81.ec2.internal
19:25:19	kserve-ci-e2e-test		isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n to ip-10-0-136-81.ec2.internal
19:21:36	kserve-ci-e2e-test		isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs to ip-10-0-139-72.ec2.internal
19:22:41	kserve-ci-e2e-test		isvc-xgboost-v2-predictor-fcc668bc-bd4kg	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bd4kg to ip-10-0-139-72.ec2.internal
19:13:01	kserve-ci-e2e-test		isvc-canary-promote-v2-predictor-85cf669bc9-ktc84	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-ktc84 to ip-10-0-139-72.ec2.internal
19:19:32	kserve-ci-e2e-test		isvc-lightgbm-predictor-54f85ddf88-574r7	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7 to ip-10-0-136-81.ec2.internal
18:58:15	kserve-ci-e2e-test		isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n to ip-10-0-139-72.ec2.internal
18:50:39	kserve-ci-e2e-test		isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc to ip-10-0-136-81.ec2.internal
19:20:11	kserve-ci-e2e-test		isvc-xgboost-runtime-predictor-54dd488c85-w4qb6	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6 to ip-10-0-136-81.ec2.internal
19:18:20	kserve-ci-e2e-test		isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d to ip-10-0-139-72.ec2.internal
19:02:00	kserve-ci-e2e-test		isvc-sklearn-predictor-7698f4f677-f9tg2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2 to ip-10-0-136-81.ec2.internal
19:03:40	kserve-ci-e2e-test		isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn to ip-10-0-136-81.ec2.internal
19:16:26	kserve-ci-e2e-test		isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw to ip-10-0-139-72.ec2.internal
19:11:20	kserve-ci-e2e-test		isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw to ip-10-0-136-81.ec2.internal
18:54:29	kserve-ci-e2e-test		isvc-logger-predictor-7656d4d74-8s8b4	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4 to ip-10-0-136-81.ec2.internal
18:50:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
18:50:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
18:50:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
18:50:39	kserve-ci-e2e-test	deployment-controller	isvc-init-fail-fce992-predictor	ScalingReplicaSet	Scaled up replica set isvc-init-fail-fce992-predictor-5f5cbffc4f from 0 to 1
18:50:39	kserve-ci-e2e-test	kubelet	isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-init-fail-fce992-predictor-serving-cert" not found
18:50:39	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v	AddedInterface	Add eth0 [10.134.0.29/23] from ovn-kubernetes
18:50:39	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-predictor-6ccc9784d9 from 0 to 1
18:50:39	kserve-ci-e2e-test	replicaset-controller	isvc-init-fail-fce992-predictor-5f5cbffc4f	SuccessfulCreate	Created pod: isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc
18:50:39	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-predictor-6ccc9784d9	SuccessfulCreate	Created pod: isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v
18:50:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v	Started	Started container storage-initializer
18:50:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v	Created	Created container: storage-initializer
18:50:40	kserve-ci-e2e-test	multus	isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc	AddedInterface	Add eth0 [10.134.0.30/23] from ovn-kubernetes
18:50:44 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
18:50:44 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc	Created	Created container: storage-initializer
18:50:44 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc	Started	Started container storage-initializer
18:50:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985"
18:50:44 (x8)	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-fce992	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:50:49	kserve-ci-e2e-test	kubelet	isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc	Killing	Stopping container storage-initializer
18:50:54	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-predictor-5f849d69df from 0 to 1
18:50:54 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn": the object has been modified; please apply your changes to the latest version and try again
18:50:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn": the object has been modified; please apply your changes to the latest version and try again
18:50:54	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-predictor-5f849d69df-zk96k	AddedInterface	Add eth0 [10.132.0.33/23] from ovn-kubernetes
18:50:54	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-predictor-5f849d69df	SuccessfulCreate	Created pod: isvc-predictive-sklearn-predictor-5f849d69df-zk96k
18:50:55	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-zk96k	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5"
18:50:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
18:50:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v	Started	Started container kserve-container
18:50:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" in 13.059s (13.059s including waiting). Image size: 1582981516 bytes.
18:50:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v	Created	Created container: kserve-container
18:50:58	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-zk96k	Started	Started container storage-initializer
18:50:58	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-zk96k	Created	Created container: storage-initializer
18:50:58	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-zk96k	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" in 3.602s (3.602s including waiting). Image size: 308610468 bytes.
18:50:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v	Pulling	Pulling image "quay.io/opendatahub/kserve-agent@sha256:03b5a032cdb9cb2925ad447181dfdd6ead3d3da3102406279a9c525a9f0d0207"
18:50:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v	Created	Created container: kube-rbac-proxy
18:50:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v	Started	Started container kube-rbac-proxy
18:50:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.213s (2.213s including waiting). Image size: 211946088 bytes.
18:51:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v	Created	Created container: agent
18:51:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:03b5a032cdb9cb2925ad447181dfdd6ead3d3da3102406279a9c525a9f0d0207" in 1.796s (1.796s including waiting). Image size: 243360842 bytes.
18:51:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v	Started	Started container agent
18:51:03	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-zk96k	Pulling	Pulling image "kserve/predictiveserver:latest"
18:51:09 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:51:09 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:51:25	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-zk96k	Pulled	Successfully pulled image "kserve/predictiveserver:latest" in 22.338s (22.338s including waiting). Image size: 2434679948 bytes.
18:51:26	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-zk96k	Created	Created container: kserve-container
18:51:26	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-zk96k	Started	Started container kserve-container
18:51:26	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-zk96k	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
18:51:28	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-zk96k	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.766s (2.766s including waiting). Image size: 211946088 bytes.
18:51:28	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-zk96k	Created	Created container: kube-rbac-proxy
18:51:28	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-zk96k	Started	Started container kube-rbac-proxy
18:52:09 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:52:09 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:52:15 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:52:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InferenceServiceReady	InferenceService [isvc-sklearn-batcher] is Ready
18:52:24 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:52:24	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-custom-predictor-8485cc5cfb	SuccessfulCreate	Created pod: isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8
18:52:24 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:52:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v	Killing	Stopping container kube-rbac-proxy
18:52:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v	Killing	Stopping container agent
18:52:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v	Killing	Stopping container kserve-container
18:52:24	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher-custom": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher-custom": the object has been modified; please apply your changes to the latest version and try again
18:52:24	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-custom-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-custom-predictor-8485cc5cfb from 0 to 1
18:52:24	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8	AddedInterface	Add eth0 [10.134.0.31/23] from ovn-kubernetes
18:52:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
18:52:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8	Created	Created container: storage-initializer
18:52:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8	Started	Started container storage-initializer
18:52:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" already present on machine
18:52:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8	Created	Created container: kserve-container
18:52:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8	Started	Started container kserve-container
18:52:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:52:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8	Created	Created container: kube-rbac-proxy
18:52:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8	Started	Started container kube-rbac-proxy
18:52:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:03b5a032cdb9cb2925ad447181dfdd6ead3d3da3102406279a9c525a9f0d0207" already present on machine
18:52:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8	Created	Created container: agent
18:52:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8	Started	Started container agent
18:52:34 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v	Unhealthy	Readiness probe failed: Get "https://10.134.0.29:8643/healthz": dial tcp 10.134.0.29:8643: connect: connection refused
18:52:35 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v	Unhealthy	Readiness probe failed: dial tcp 10.134.0.29:8080: connect: connection refused
18:52:35 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
18:52:39	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:52:39	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:52:46 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:52:46	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	InferenceServiceReady	InferenceService [isvc-predictive-sklearn] is Ready
18:52:54	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-zk96k	Killing	Stopping container kserve-container
18:52:54	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-zk96k	Killing	Stopping container kube-rbac-proxy
18:52:56 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-zk96k	Unhealthy	Readiness probe failed: dial tcp 10.132.0.33:8080: connect: connection refused
18:52:56	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-zk96k	Unhealthy	Readiness probe failed: Get "https://10.132.0.33:8643/healthz": dial tcp 10.132.0.33:8643: connect: connection refused
18:52:59	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-predictor-695f978978	SuccessfulCreate	Created pod: isvc-predictive-xgboost-predictor-695f978978-cp8h8
18:52:59	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost": the object has been modified; please apply your changes to the latest version and try again
18:52:59	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-predictor-695f978978 from 0 to 1
18:53:00	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-cp8h8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
18:53:00	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-cp8h8	Created	Created container: storage-initializer
18:53:00	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-cp8h8	Started	Started container storage-initializer
18:53:00	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-predictor-695f978978-cp8h8	AddedInterface	Add eth0 [10.134.0.32/23] from ovn-kubernetes
18:53:05	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-cp8h8	Pulling	Pulling image "kserve/predictiveserver:latest"
18:53:09 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:53:09 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:53:27	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-cp8h8	Pulled	Successfully pulled image "kserve/predictiveserver:latest" in 22.242s (22.242s including waiting). Image size: 2434679947 bytes.
18:53:27	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-cp8h8	Created	Created container: kserve-container
18:53:27	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-cp8h8	Started	Started container kube-rbac-proxy
18:53:27	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-cp8h8	Started	Started container kserve-container
18:53:27	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-cp8h8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:53:27	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-cp8h8	Created	Created container: kube-rbac-proxy
18:53:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	InferenceServiceReady	InferenceService [isvc-sklearn-batcher-custom] is Ready
18:53:55 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:53:59 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:53:59 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:54:09	kserve-ci-e2e-test	replicaset-controller	message-dumper-predictor-6575bcfdfd	SuccessfulCreate	Created pod: message-dumper-predictor-6575bcfdfd-c6scw
18:54:09	kserve-ci-e2e-test	deployment-controller	message-dumper-predictor	ScalingReplicaSet	Scaled up replica set message-dumper-predictor-6575bcfdfd from 0 to 1
18:54:09 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:54:09 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:54:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8	Killing	Stopping container kube-rbac-proxy
18:54:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8	Killing	Stopping container agent
18:54:09	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	UpdateFailed	Failed to update status for InferenceService "message-dumper": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper": the object has been modified; please apply your changes to the latest version and try again
18:54:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8	Killing	Stopping container kserve-container
18:54:10	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-c6scw	Pulling	Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display"
18:54:10	kserve-ci-e2e-test	multus	message-dumper-predictor-6575bcfdfd-c6scw	AddedInterface	Add eth0 [10.132.0.34/23] from ovn-kubernetes
18:54:11	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-c6scw	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:54:11	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-c6scw	Started	Started container kserve-container
18:54:11	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-c6scw	Pulled	Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.227s (1.227s including waiting). Image size: 14813193 bytes.
18:54:11	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-c6scw	Started	Started container kube-rbac-proxy
18:54:11	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-c6scw	Created	Created container: kube-rbac-proxy
18:54:11	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-c6scw	Created	Created container: kserve-container
18:54:15 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
18:54:15 (x12)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8	Unhealthy	Readiness probe failed: dial tcp 10.134.0.31:5000: connect: connection refused
18:54:16 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8	Unhealthy	Readiness probe failed: Get "https://10.134.0.31:8643/healthz": dial tcp 10.134.0.31:8643: connect: connection refused
18:54:19 (x7)	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:54:19	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	InferenceServiceReady	InferenceService [message-dumper] is Ready
18:54:29	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7656d4d74-8s8b4	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-logger-predictor-serving-cert" not found
18:54:29	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	UpdateFailed	Failed to update status for InferenceService "isvc-logger": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger": the object has been modified; please apply your changes to the latest version and try again
18:54:29	kserve-ci-e2e-test	deployment-controller	isvc-logger-predictor	ScalingReplicaSet	Scaled up replica set isvc-logger-predictor-7656d4d74 from 0 to 1
18:54:29	kserve-ci-e2e-test	replicaset-controller	isvc-logger-predictor-7656d4d74	SuccessfulCreate	Created pod: isvc-logger-predictor-7656d4d74-8s8b4
18:54:30	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7656d4d74-8s8b4	Created	Created container: storage-initializer
18:54:30	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7656d4d74-8s8b4	Started	Started container storage-initializer
18:54:30	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7656d4d74-8s8b4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
18:54:30	kserve-ci-e2e-test	multus	isvc-logger-predictor-7656d4d74-8s8b4	AddedInterface	Add eth0 [10.134.0.33/23] from ovn-kubernetes
18:54:35	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7656d4d74-8s8b4	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:03b5a032cdb9cb2925ad447181dfdd6ead3d3da3102406279a9c525a9f0d0207" already present on machine
18:54:35 (x9)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-cp8h8	Unhealthy	Readiness probe failed: dial tcp 10.134.0.32:8080: connect: connection refused
18:54:35	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7656d4d74-8s8b4	Started	Started container kube-rbac-proxy
18:54:35	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7656d4d74-8s8b4	Created	Created container: kube-rbac-proxy
18:54:35	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7656d4d74-8s8b4	Created	Created container: agent
18:54:35	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7656d4d74-8s8b4	Started	Started container agent
18:54:35	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7656d4d74-8s8b4	Created	Created container: kserve-container
18:54:35	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7656d4d74-8s8b4	Started	Started container kserve-container
18:54:35	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7656d4d74-8s8b4	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" already present on machine
18:54:35	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7656d4d74-8s8b4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:54:44 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:54:44 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:54:45 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:54:45	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	InferenceServiceReady	InferenceService [isvc-predictive-xgboost] is Ready
18:54:49	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-cp8h8	Killing	Stopping container kserve-container
18:54:49	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-cp8h8	Killing	Stopping container kube-rbac-proxy
18:54:50	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-cp8h8	Unhealthy	Readiness probe failed: Get "https://10.134.0.32:8643/healthz": dial tcp 10.134.0.32:8643: connect: connection refused
18:54:54	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-predictor-ff56bcbb5 from 0 to 1
18:54:54	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-predictor-ff56bcbb5	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx
18:54:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm": the object has been modified; please apply your changes to the latest version and try again
18:54:55	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
18:54:56	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx	Created	Created container: storage-initializer
18:54:56	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx	AddedInterface	Add eth0 [10.134.0.34/23] from ovn-kubernetes
18:54:56	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx	Started	Started container storage-initializer
18:54:56	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
18:54:59 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:54:59 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:55:00	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx	Started	Started container kserve-container
18:55:00	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
18:55:00	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx	Created	Created container: kserve-container
18:55:00	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:55:00	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx	Created	Created container: kube-rbac-proxy
18:55:00	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx	Started	Started container kube-rbac-proxy
18:55:09 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:55:09 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:55:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:55:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:55:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	InferenceServiceReady	InferenceService [isvc-logger] is Ready
18:55:53 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:55:59 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:55:59 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:56:04	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v1-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v1-predictor-574b5b7495 from 0 to 1
18:56:04	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v1-predictor-574b5b7495	SuccessfulCreate	Created pod: isvc-autogluon-v1-predictor-574b5b7495-r5d6n
18:56:04	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7656d4d74-8s8b4	Killing	Stopping container kserve-container
18:56:04	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7656d4d74-8s8b4	Killing	Stopping container agent
18:56:04	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-c6scw	Killing	Stopping container kserve-container
18:56:04	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-c6scw	Killing	Stopping container kube-rbac-proxy
18:56:04	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7656d4d74-8s8b4	Killing	Stopping container kube-rbac-proxy
18:56:05	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-r5d6n	Started	Started container storage-initializer
18:56:05	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-r5d6n	Created	Created container: storage-initializer
18:56:05	kserve-ci-e2e-test	multus	isvc-autogluon-v1-predictor-574b5b7495-r5d6n	AddedInterface	Add eth0 [10.134.0.35/23] from ovn-kubernetes
18:56:05	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-r5d6n	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
18:56:13 (x11)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7656d4d74-8s8b4	Unhealthy	Readiness probe failed: dial tcp 10.134.0.33:8080: connect: connection refused
18:56:13 (x10)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7656d4d74-8s8b4	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
18:56:18 (x3)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7656d4d74-8s8b4	Unhealthy	Readiness probe failed: Get "https://10.134.0.33:8643/healthz": dial tcp 10.134.0.33:8643: connect: connection refused
18:56:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm] is Ready
18:56:18 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:56:24	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx	Killing	Stopping container kube-rbac-proxy
18:56:24 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:56:24	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx	Killing	Stopping container kserve-container
18:56:24 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:56:28 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx	Unhealthy	Readiness probe failed: dial tcp 10.134.0.34:8080: connect: connection refused
18:56:28	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx	Unhealthy	Readiness probe failed: Get "https://10.134.0.34:8643/healthz": dial tcp 10.134.0.34:8643: connect: connection refused
18:56:30	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf	AddedInterface	Add eth0 [10.132.0.35/23] from ovn-kubernetes
18:56:30	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
18:56:30	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-v2-predictor-745f9c8c9f	SuccessfulCreate	Created pod: isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf
18:56:30	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-v2-predictor-745f9c8c9f from 0 to 1
18:56:30	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf	Started	Started container storage-initializer
18:56:30	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf	Created	Created container: storage-initializer
18:56:30	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
18:56:35	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:56:35	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf	Created	Created container: kube-rbac-proxy
18:56:35	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf	Started	Started container kube-rbac-proxy
18:56:35	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf	Started	Started container kserve-container
18:56:35	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf	Created	Created container: kserve-container
18:56:35	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
18:56:36	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-r5d6n	Pulling	Pulling image "kserve/autogluonserver:latest"
18:57:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:57:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:58:00 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:58:00 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:58:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	InferenceServiceReady	InferenceService [isvc-predictive-sklearn-v2] is Ready
18:58:03 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:58:10	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf	Killing	Stopping container kube-rbac-proxy
18:58:10	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf	Killing	Stopping container kserve-container
18:58:13 (x6)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf	Unhealthy	Readiness probe failed: Get "http://10.132.0.35:8080/v2/models/isvc-predictive-sklearn-v2/ready": dial tcp 10.132.0.35:8080: connect: connection refused
18:58:13	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ldkkf	Unhealthy	Readiness probe failed: Get "https://10.132.0.35:8643/healthz": dial tcp 10.132.0.35:8643: connect: connection refused
18:58:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
18:58:15	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-v2-predictor-868ff5dd6d	SuccessfulCreate	Created pod: isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n
18:58:15	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
18:58:15	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-v2-predictor-868ff5dd6d from 0 to 1
18:58:16	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
18:58:16	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n	AddedInterface	Add eth0 [10.132.0.36/23] from ovn-kubernetes
18:58:16	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n	Started	Started container storage-initializer
18:58:16	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n	Created	Created container: storage-initializer
18:58:20	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:58:20	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n	Started	Started container kube-rbac-proxy
18:58:20	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n	Created	Created container: kserve-container
18:58:20	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
18:58:20	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n	Started	Started container kserve-container
18:58:20	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n	Created	Created container: kube-rbac-proxy
18:58:37	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-r5d6n	Pulled	Successfully pulled image "kserve/autogluonserver:latest" in 2m1.767s (2m1.767s including waiting). Image size: 8945772325 bytes.
18:58:38	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-r5d6n	Started	Started container kserve-container
18:58:38	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-r5d6n	Started	Started container kube-rbac-proxy
18:58:38	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-r5d6n	Created	Created container: kserve-container
18:58:38	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-r5d6n	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:58:38	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-r5d6n	Created	Created container: kube-rbac-proxy
18:58:44 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-r5d6n	Unhealthy	Readiness probe failed: dial tcp 10.134.0.35:8080: connect: connection refused
18:58:45 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:58:45 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:58:49 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:58:49 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:58:54 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:58:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	InferenceServiceReady	InferenceService [isvc-autogluon-v1] is Ready
18:59:04	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:59:04	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:59:11	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-r5d6n	Killing	Stopping container kserve-container
18:59:11	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-r5d6n	Killing	Stopping container kube-rbac-proxy
18:59:16	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-predictor-9684b4cd9	SuccessfulCreate	Created pod: isvc-autogluon-v2-predictor-9684b4cd9-wps2s
18:59:16	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-predictor-9684b4cd9 from 0 to 1
18:59:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2": the object has been modified; please apply your changes to the latest version and try again
18:59:17	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-wps2s	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
18:59:17	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-wps2s	Created	Created container: storage-initializer
18:59:17	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-wps2s	Started	Started container storage-initializer
18:59:17	kserve-ci-e2e-test	multus	isvc-autogluon-v2-predictor-9684b4cd9-wps2s	AddedInterface	Add eth0 [10.134.0.36/23] from ovn-kubernetes
18:59:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	InferenceServiceReady	InferenceService [isvc-predictive-xgboost-v2] is Ready
18:59:38 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:59:45	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n	Killing	Stopping container kube-rbac-proxy
18:59:45 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:59:45 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:59:45	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n	Killing	Stopping container kserve-container
18:59:47	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-wps2s	Created	Created container: kserve-container
18:59:47	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-wps2s	Started	Started container kube-rbac-proxy
18:59:47	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-wps2s	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
18:59:47	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-wps2s	Created	Created container: kube-rbac-proxy
18:59:47	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-wps2s	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:59:47	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-wps2s	Started	Started container kserve-container
18:59:48 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n	Unhealthy	Readiness probe failed: Get "http://10.132.0.36:8080/v2/models/isvc-predictive-xgboost-v2/ready": dial tcp 10.132.0.36:8080: connect: connection refused
18:59:48	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-cpx7n	Unhealthy	Readiness probe failed: Get "https://10.132.0.36:8643/healthz": dial tcp 10.132.0.36:8643: connect: connection refused
18:59:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm-v2": the object has been modified; please apply your changes to the latest version and try again
18:59:50	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-v2-predictor-545b7cc587	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565
18:59:50	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-v2-predictor-545b7cc587 from 0 to 1
18:59:51	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565	AddedInterface	Add eth0 [10.132.0.37/23] from ovn-kubernetes
18:59:51	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
18:59:51	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565	Created	Created container: storage-initializer
18:59:51	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565	Started	Started container storage-initializer
18:59:56	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
18:59:56	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565	Started	Started container kube-rbac-proxy
18:59:56	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565	Created	Created container: kube-rbac-proxy
18:59:56	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565	Created	Created container: kserve-container
18:59:56	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565	Started	Started container kserve-container
18:59:56	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:00:16 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:00:16 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:00:35 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:00:35 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:01:14 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:01:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm-v2] is Ready
19:01:20 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:01:20 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:01:20	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565	Killing	Stopping container kube-rbac-proxy
19:01:20	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565	Killing	Stopping container kserve-container
19:01:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	InferenceServiceReady	InferenceService [isvc-autogluon-v2] is Ready
19:01:22 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:01:24	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565	Unhealthy	Readiness probe failed: Get "https://10.132.0.37:8643/healthz": dial tcp 10.132.0.37:8643: connect: connection refused
19:01:24 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-26565	Unhealthy	Readiness probe failed: Get "http://10.132.0.37:8080/v2/models/isvc-predictive-lightgbm-v2/ready": dial tcp 10.132.0.37:8080: connect: connection refused
19:01:25	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-scheduler": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
19:01:25	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
19:01:25	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-scheduler-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-scheduler-predictor-58b5cb5497 from 0 to 1
19:01:25	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-scheduler-predictor-58b5cb5497	SuccessfulCreate	Created pod: isvc-sklearn-scheduler-predictor-58b5cb5497-ttdbv
19:01:26 (x5)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:01:31 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:01:31 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:01:32	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-wps2s	Killing	Stopping container kserve-container
19:01:32	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-wps2s	Killing	Stopping container kube-rbac-proxy
19:01:38	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-variants-predictor-5c889d8844	SuccessfulCreate	Created pod: isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8
19:01:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2-variants": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-variants": the object has been modified; please apply your changes to the latest version and try again
19:01:38	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-variants-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-variants-predictor-5c889d8844 from 0 to 1
19:01:38	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
19:01:39	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8	Created	Created container: storage-initializer
19:01:39	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:01:39	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8	Started	Started container storage-initializer
19:01:39	kserve-ci-e2e-test	multus	isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8	AddedInterface	Add eth0 [10.134.0.37/23] from ovn-kubernetes
19:01:44	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:01:44	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8	Started	Started container kube-rbac-proxy
19:01:44	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8	Created	Created container: kube-rbac-proxy
19:01:44	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8	Started	Started container kserve-container
19:01:44	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8	Created	Created container: kserve-container
19:01:44	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
19:02:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn": the object has been modified; please apply your changes to the latest version and try again
19:02:00	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-predictor-7698f4f677	SuccessfulCreate	Created pod: isvc-sklearn-predictor-7698f4f677-f9tg2
19:02:00	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-predictor-7698f4f677 from 0 to 1
19:02:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7698f4f677-f9tg2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:02:01	kserve-ci-e2e-test	multus	isvc-sklearn-predictor-7698f4f677-f9tg2	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
19:02:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7698f4f677-f9tg2	Created	Created container: storage-initializer
19:02:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7698f4f677-f9tg2	Started	Started container storage-initializer
19:02:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7698f4f677-f9tg2	Started	Started container kube-rbac-proxy
19:02:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7698f4f677-f9tg2	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" already present on machine
19:02:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7698f4f677-f9tg2	Created	Created container: kserve-container
19:02:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7698f4f677-f9tg2	Started	Started container kserve-container
19:02:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7698f4f677-f9tg2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:02:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7698f4f677-f9tg2	Created	Created container: kube-rbac-proxy
19:02:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:02:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:02:30 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:02:30 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:03:19 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:03:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	InferenceServiceReady	InferenceService [isvc-autogluon-v2-variants] is Ready
19:03:23 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:03:23 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:03:24 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:03:24	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	InferenceServiceReady	InferenceService [isvc-sklearn] is Ready
19:03:30 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:03:30	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8	Unhealthy	Readiness probe failed: Get "http://10.134.0.37:8080/v2/models/isvc-autogluon-v2-variants/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
19:03:30 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:03:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7698f4f677-f9tg2	Killing	Stopping container kube-rbac-proxy
19:03:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7698f4f677-f9tg2	Killing	Stopping container kserve-container
19:03:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7698f4f677-f9tg2	Unhealthy	Readiness probe failed: Get "https://10.134.0.38:8643/healthz": dial tcp 10.134.0.38:8643: connect: connection refused
19:03:34	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8	Killing	Stopping container kube-rbac-proxy
19:03:34	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8	Killing	Stopping container kserve-container
19:03:34 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7698f4f677-f9tg2	Unhealthy	Readiness probe failed: dial tcp 10.134.0.38:8080: connect: connection refused
19:03:36	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:03:36	kserve-ci-e2e-test	deployment-controller	sklearn-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set sklearn-v2-mlserver-predictor-7f568cdc6b from 0 to 1
19:03:36	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl	Created	Created container: storage-initializer
19:03:36	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "sklearn-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
19:03:36 (x2)	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "sklearn-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "sklearn-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
19:03:36	kserve-ci-e2e-test	multus	sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl	AddedInterface	Add eth0 [10.132.0.38/23] from ovn-kubernetes
19:03:36	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl	Started	Started container storage-initializer
19:03:36	kserve-ci-e2e-test	replicaset-controller	sklearn-v2-mlserver-predictor-7f568cdc6b	SuccessfulCreate	Created pod: sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl
19:03:40	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn	Started	Started container storage-initializer
19:03:40	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:03:40	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-noslash-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-noslash-predictor-5499c79f9f from 0 to 1
19:03:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2-noslash": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-noslash": the object has been modified; please apply your changes to the latest version and try again
19:03:40	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-noslash-predictor-5499c79f9f	SuccessfulCreate	Created pod: isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn
19:03:40	kserve-ci-e2e-test	multus	isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
19:03:40	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn	Created	Created container: storage-initializer
19:03:41	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl	Pulling	Pulling image "docker.io/seldonio/mlserver:1.7.1"
19:04:10	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
19:04:10	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:04:10	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn	Created	Created container: kube-rbac-proxy
19:04:10	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn	Started	Started container kube-rbac-proxy
19:04:10	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn	Started	Started container kserve-container
19:04:10	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn	Created	Created container: kserve-container
19:04:40 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:04:40 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:05:41 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:05:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	InferenceServiceReady	InferenceService [isvc-autogluon-v2-noslash] is Ready
19:05:52	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn	Unhealthy	Readiness probe failed: Get "http://10.134.0.39:8080/v2/models/isvc-autogluon-v2-noslash/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
19:05:55 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:05:55 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:05:56	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl	Pulled	Successfully pulled image "docker.io/seldonio/mlserver:1.7.1" in 2m15.756s (2m15.756s including waiting). Image size: 10890461297 bytes.
19:05:56	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl	Created	Created container: kserve-container
19:05:56	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn	Killing	Stopping container kserve-container
19:05:56	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn	Killing	Stopping container kube-rbac-proxy
19:05:57	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl	Started	Started container kserve-container
19:05:57	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:05:57	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl	Created	Created container: kube-rbac-proxy
19:05:57	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl	Started	Started container kube-rbac-proxy
19:06:01	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-ts-v1-predictor-6cf79d9f7d	SuccessfulCreate	Created pod: isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp
19:06:01	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-ts-v1-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-ts-v1-predictor-6cf79d9f7d from 0 to 1
19:06:01 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-ts-v1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1": the object has been modified; please apply your changes to the latest version and try again
19:06:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1": the object has been modified; please apply your changes to the latest version and try again
19:06:01	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
19:06:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp	Started	Started container storage-initializer
19:06:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp	Created	Created container: storage-initializer
19:06:03	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:06:03	kserve-ci-e2e-test	multus	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp	AddedInterface	Add eth0 [10.132.0.39/23] from ovn-kubernetes
19:06:06 (x10)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:06:06 (x10)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:06:08	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp	Pulling	Pulling image "kserve/autogluonserver:latest"
19:06:36 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:06:44 (x2)	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
19:06:51 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:06:54	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	InferenceServiceReady	InferenceService [sklearn-v2-mlserver] is Ready
19:06:55 (x12)	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:06:56	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl	Killing	Stopping container kserve-container
19:06:56	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl	Killing	Stopping container kube-rbac-proxy
19:07:01	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-runtime-predictor-b78d6df9c	SuccessfulCreate	Created pod: isvc-sklearn-runtime-predictor-b78d6df9c-vmwll
19:07:01	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-runtime-predictor-b78d6df9c from 0 to 1
19:07:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-runtime": the object has been modified; please apply your changes to the latest version and try again
19:07:02	kserve-ci-e2e-test	multus	isvc-sklearn-runtime-predictor-b78d6df9c-vmwll	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
19:07:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-b78d6df9c-vmwll	Started	Started container storage-initializer
19:07:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-b78d6df9c-vmwll	Created	Created container: storage-initializer
19:07:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-b78d6df9c-vmwll	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:07:04 (x2)	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl	Unhealthy	Readiness probe failed: Get "https://10.132.0.38:8643/healthz": dial tcp 10.132.0.38:8643: connect: connection refused
19:07:04	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-s8ksl	Unhealthy	Readiness probe failed: Get "http://10.132.0.38:8080/v2/models/sklearn-v2-mlserver/ready": dial tcp 10.132.0.38:8080: connect: connection refused
19:07:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-b78d6df9c-vmwll	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" already present on machine
19:07:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-b78d6df9c-vmwll	Started	Started container kube-rbac-proxy
19:07:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-b78d6df9c-vmwll	Created	Created container: kube-rbac-proxy
19:07:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-b78d6df9c-vmwll	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:07:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-b78d6df9c-vmwll	Started	Started container kserve-container
19:07:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-b78d6df9c-vmwll	Created	Created container: kserve-container
19:07:15 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-b78d6df9c-vmwll	Unhealthy	Readiness probe failed: dial tcp 10.134.0.40:8080: connect: connection refused
19:07:25 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:07:25	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-runtime] is Ready
19:07:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:07:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:07:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-b78d6df9c-vmwll	Killing	Stopping container kube-rbac-proxy
19:07:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-b78d6df9c-vmwll	Killing	Stopping container kserve-container
19:07:38 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
19:07:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
19:07:38	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-runtime-predictor-5f69c7d498 from 0 to 1
19:07:38	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-runtime-predictor-5f69c7d498	SuccessfulCreate	Created pod: isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h
19:08:10	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:08:10	kserve-ci-e2e-test	multus	isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h	AddedInterface	Add eth0 [10.132.0.40/23] from ovn-kubernetes
19:08:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h	Started	Started container storage-initializer
19:08:11	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp	Created	Created container: kserve-container
19:08:11	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp	Started	Started container kube-rbac-proxy
19:08:11	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp	Created	Created container: kube-rbac-proxy
19:08:11	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:08:11	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp	Started	Started container kserve-container
19:08:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h	Created	Created container: storage-initializer
19:08:11	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp	Pulled	Successfully pulled image "kserve/autogluonserver:latest" in 2m2.283s (2m2.283s including waiting). Image size: 8945772325 bytes.
19:08:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:08:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h	Started	Started container kube-rbac-proxy
19:08:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h	Started	Started container kserve-container
19:08:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h	Created	Created container: kube-rbac-proxy
19:08:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h	Created	Created container: kserve-container
19:08:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:08:16 (x9)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:08:16 (x9)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:08:27	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	InferenceServiceReady	InferenceService [isvc-autogluon-ts-v1] is Ready
19:08:27 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:08:31	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:08:31	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:08:45	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp	Killing	Stopping container kserve-container
19:08:45	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp	Killing	Stopping container kube-rbac-proxy
19:08:47	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp	Unhealthy	Readiness probe failed: Get "https://10.132.0.39:8643/healthz": dial tcp 10.132.0.39:8643: connect: connection refused
19:08:47 (x4)	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-mbczp	Unhealthy	Readiness probe failed: dial tcp 10.132.0.39:8080: connect: connection refused
19:08:50	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-ts-v1-noslash-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-ts-v1-noslash-predictor-b54fbf585 from 0 to 1
19:08:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-ts-v1-noslash": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1-noslash": the object has been modified; please apply your changes to the latest version and try again
19:08:50	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
19:08:50	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585	SuccessfulCreate	Created pod: isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8
19:08:51	kserve-ci-e2e-test	multus	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8	AddedInterface	Add eth0 [10.132.0.41/23] from ovn-kubernetes
19:08:51	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:08:51	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8	Started	Started container storage-initializer
19:08:51	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8	Created	Created container: storage-initializer
19:08:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
19:08:53 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:08:53 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:09:01	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8	Created	Created container: kube-rbac-proxy
19:09:01	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:09:01	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8	Started	Started container kserve-container
19:09:01	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8	Created	Created container: kserve-container
19:09:01	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
19:09:02	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8	Started	Started container kube-rbac-proxy
19:09:02	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-v2-runtime] is Ready
19:09:02 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:09:08 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:09:08	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedUpdateStatus	horizontalpodautoscalers.autoscaling "isvc-sklearn-v2-runtime-predictor" not found
19:09:08 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:09:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h	Killing	Stopping container kserve-container
19:09:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h	Killing	Stopping container kube-rbac-proxy
19:09:09 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8	Unhealthy	Readiness probe failed: dial tcp 10.132.0.41:8080: connect: connection refused
19:09:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h	Unhealthy	Readiness probe failed: Get "https://10.132.0.40:8643/healthz": dial tcp 10.132.0.40:8643: connect: connection refused
19:09:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-gxz5h	Unhealthy	Readiness probe failed: Get "http://10.132.0.40:8080/v2/models/isvc-sklearn-v2-runtime/ready": dial tcp 10.132.0.40:8080: connect: connection refused
19:09:13	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-predictor-5fd47fbf89	SuccessfulCreate	Created pod: isvc-sklearn-v2-predictor-5fd47fbf89-w5j95
19:09:13	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
19:09:13	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-predictor-5fd47fbf89 from 0 to 1
19:09:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5fd47fbf89-w5j95	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-v2-predictor-serving-cert" not found
19:09:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5fd47fbf89-w5j95	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:09:15	kserve-ci-e2e-test	multus	isvc-sklearn-v2-predictor-5fd47fbf89-w5j95	AddedInterface	Add eth0 [10.134.0.41/23] from ovn-kubernetes
19:09:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5fd47fbf89-w5j95	Started	Started container storage-initializer
19:09:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5fd47fbf89-w5j95	Created	Created container: storage-initializer
19:09:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5fd47fbf89-w5j95	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:09:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5fd47fbf89-w5j95	Started	Started container kube-rbac-proxy
19:09:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5fd47fbf89-w5j95	Created	Created container: kube-rbac-proxy
19:09:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5fd47fbf89-w5j95	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" already present on machine
19:09:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5fd47fbf89-w5j95	Created	Created container: kserve-container
19:09:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5fd47fbf89-w5j95	Started	Started container kserve-container
19:09:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	InferenceServiceReady	InferenceService [isvc-autogluon-ts-v1-noslash] is Ready
19:09:20 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:09:20 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:09:20 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:09:28	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:09:28	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:09:30	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8	Killing	Stopping container kube-rbac-proxy
19:09:30	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bn9f8	Killing	Stopping container kserve-container
19:09:35	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-v2-predictor-6c79c974bf	SuccessfulCreate	Created pod: isvc-canary-create-v2-predictor-6c79c974bf-bdhcj
19:09:35	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-zvdx9	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-create-predictor-serving-cert" not found
19:09:35	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-68cd88b8	SuccessfulCreate	Created pod: isvc-canary-create-predictor-68cd88b8-vwczt
19:09:35	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-create-predictor-68cd88b8 from 0 to 2
19:09:35	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-create-v2-predictor-6c79c974bf from 0 to 1
19:09:35	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-68cd88b8	SuccessfulCreate	Created pod: isvc-canary-create-predictor-68cd88b8-zvdx9
19:09:36	kserve-ci-e2e-test	multus	isvc-canary-create-v2-predictor-6c79c974bf-bdhcj	AddedInterface	Add eth0 [10.134.0.42/23] from ovn-kubernetes
19:09:36	kserve-ci-e2e-test	multus	isvc-canary-create-predictor-68cd88b8-vwczt	AddedInterface	Add eth0 [10.132.0.42/23] from ovn-kubernetes
19:09:36	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6c79c974bf-bdhcj	Started	Started container storage-initializer
19:09:36	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6c79c974bf-bdhcj	Created	Created container: storage-initializer
19:09:36	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6c79c974bf-bdhcj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:09:36	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-vwczt	Created	Created container: storage-initializer
19:09:36	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-vwczt	Started	Started container storage-initializer
19:09:36	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-vwczt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:09:36	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-zvdx9	Started	Started container storage-initializer
19:09:36	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-zvdx9	Created	Created container: storage-initializer
19:09:36	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-zvdx9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:09:36	kserve-ci-e2e-test	multus	isvc-canary-create-predictor-68cd88b8-zvdx9	AddedInterface	Add eth0 [10.134.0.43/23] from ovn-kubernetes
19:09:40	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-zvdx9	Created	Created container: kserve-container
19:09:40	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6c79c974bf-bdhcj	Started	Started container kube-rbac-proxy
19:09:40	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-zvdx9	Started	Started container kserve-container
19:09:40	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-zvdx9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:09:40	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-zvdx9	Created	Created container: kube-rbac-proxy
19:09:40	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-zvdx9	Started	Started container kube-rbac-proxy
19:09:40	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-zvdx9	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" already present on machine
19:09:40	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6c79c974bf-bdhcj	Created	Created container: kserve-container
19:09:40	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-vwczt	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985"
19:09:40	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6c79c974bf-bdhcj	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" already present on machine
19:09:40	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6c79c974bf-bdhcj	Created	Created container: kube-rbac-proxy
19:09:40	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6c79c974bf-bdhcj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:09:40	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6c79c974bf-bdhcj	Started	Started container kserve-container
19:09:53	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-vwczt	Started	Started container kserve-container
19:09:53	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-vwczt	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:09:53	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-vwczt	Created	Created container: kserve-container
19:09:53	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-vwczt	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" in 13.373s (13.373s including waiting). Image size: 1582981516 bytes.
19:09:53	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-vwczt	Created	Created container: kube-rbac-proxy
19:09:53	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-vwczt	Started	Started container kube-rbac-proxy
19:10:37	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	InferenceServiceReady	InferenceService [isvc-sklearn-v2] is Ready
19:10:37 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:10:43 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:10:43 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:10:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5fd47fbf89-w5j95	Killing	Stopping container kube-rbac-proxy
19:10:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5fd47fbf89-w5j95	Killing	Stopping container kserve-container
19:10:47 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5fd47fbf89-w5j95	Unhealthy	Readiness probe failed: dial tcp 10.134.0.41:8080: connect: connection refused
19:10:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5fd47fbf89-w5j95	Unhealthy	Readiness probe failed: Get "https://10.134.0.41:8643/healthz": dial tcp 10.134.0.41:8643: connect: connection refused
19:10:49	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-mixed-predictor-64bd78979b	SuccessfulCreate	Created pod: isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj
19:10:49	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-mixed-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-mixed-predictor-64bd78979b from 0 to 1
19:10:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-mixed": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-mixed": the object has been modified; please apply your changes to the latest version and try again
19:10:50 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
19:10:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj	Created	Created container: storage-initializer
19:10:51 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-zvdx9	Unhealthy	Readiness probe failed: dial tcp 10.134.0.43:8080: connect: connection refused
19:10:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj	Started	Started container storage-initializer
19:10:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:10:51	kserve-ci-e2e-test	multus	isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj	AddedInterface	Add eth0 [10.132.0.43/23] from ovn-kubernetes
19:10:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" already present on machine
19:10:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj	Created	Created container: kserve-container
19:10:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj	Started	Started container kserve-container
19:10:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:10:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj	Created	Created container: kube-rbac-proxy
19:10:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj	Started	Started container kube-rbac-proxy
19:10:57 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	UpdateFailed	Failed to update status for InferenceService "isvc-canary-create": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again
19:10:57	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-zvdx9	Killing	Stopping container kube-rbac-proxy
19:10:57	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-zvdx9	Killing	Stopping container kserve-container
19:10:57	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-68cd88b8	SuccessfulDelete	Deleted pod: isvc-canary-create-predictor-68cd88b8-zvdx9
19:10:57 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again
19:10:57	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-create-predictor-68cd88b8 from 2 to 1
19:10:58	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-zvdx9	Unhealthy	Readiness probe failed: Get "https://10.134.0.43:8643/healthz": dial tcp 10.134.0.43:8643: connect: connection refused
19:11:01 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-vwczt	Unhealthy	Readiness probe failed: dial tcp 10.132.0.42:8080: connect: connection refused
19:11:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	InferenceServiceReady	InferenceService [isvc-canary-create] is Ready
19:11:11 (x21)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:11:15	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6c79c974bf-bdhcj	Killing	Stopping container kserve-container
19:11:15	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-vwczt	Killing	Stopping container kserve-container
19:11:15	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6c79c974bf-bdhcj	Killing	Stopping container kube-rbac-proxy
19:11:15	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-vwczt	Killing	Stopping container kube-rbac-proxy
19:11:16	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-vwczt	Unhealthy	Readiness probe failed: Get "https://10.132.0.42:8643/healthz": dial tcp 10.132.0.42:8643: connect: connection refused
19:11:17 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6c79c974bf-bdhcj	Unhealthy	Readiness probe failed: dial tcp 10.134.0.42:8080: connect: connection refused
19:11:17	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-6c79c974bf-bdhcj	Unhealthy	Readiness probe failed: Get "https://10.134.0.42:8643/healthz": dial tcp 10.134.0.42:8643: connect: connection refused
19:11:20	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-v2-predictor-85cf669bc9	SuccessfulCreate	Created pod: isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw
19:11:20	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-v2-predictor-85cf669bc9 from 0 to 1
19:11:20	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-predictor-674955ff8d from 0 to 2
19:11:20	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-674955ff8d	SuccessfulCreate	Created pod: isvc-canary-promote-predictor-674955ff8d-ns5mk
19:11:20	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-674955ff8d	SuccessfulCreate	Created pod: isvc-canary-promote-predictor-674955ff8d-njhv9
19:11:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	UpdateFailed	Failed to update status for InferenceService "isvc-canary-promote": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again
19:11:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again
19:11:21	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:11:21	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw	Started	Started container storage-initializer
19:11:21	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-njhv9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:11:21	kserve-ci-e2e-test	multus	isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw	AddedInterface	Add eth0 [10.134.0.44/23] from ovn-kubernetes
19:11:21	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-ns5mk	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:11:21	kserve-ci-e2e-test	multus	isvc-canary-promote-predictor-674955ff8d-ns5mk	AddedInterface	Add eth0 [10.134.0.45/23] from ovn-kubernetes
19:11:21	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw	Created	Created container: storage-initializer
19:11:21	kserve-ci-e2e-test	multus	isvc-canary-promote-predictor-674955ff8d-njhv9	AddedInterface	Add eth0 [10.132.0.44/23] from ovn-kubernetes
19:11:21	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-ns5mk	Created	Created container: storage-initializer
19:11:21	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-njhv9	Created	Created container: storage-initializer
19:11:21	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-ns5mk	Started	Started container storage-initializer
19:11:21	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-njhv9	Started	Started container storage-initializer
19:11:22	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-vwczt	Unhealthy	Readiness probe failed: dial tcp 10.132.0.42:8080: i/o timeout
19:11:22	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-68cd88b8-vwczt	Unhealthy	Readiness probe failed: Get "https://10.132.0.42:8643/healthz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
19:11:25	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw	Created	Created container: kserve-container
19:11:25	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw	Started	Started container kserve-container
19:11:25	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw	Created	Created container: kube-rbac-proxy
19:11:25	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:11:25	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-njhv9	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" already present on machine
19:11:25	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-njhv9	Created	Created container: kserve-container
19:11:25	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-njhv9	Started	Started container kserve-container
19:11:25	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-njhv9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:11:25	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-njhv9	Created	Created container: kube-rbac-proxy
19:11:25	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw	Started	Started container kube-rbac-proxy
19:11:25	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-njhv9	Started	Started container kube-rbac-proxy
19:11:25	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" already present on machine
19:11:26	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-ns5mk	Created	Created container: kube-rbac-proxy
19:11:26	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-ns5mk	Created	Created container: kserve-container
19:11:26	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-ns5mk	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" already present on machine
19:11:26	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-ns5mk	Started	Started container kserve-container
19:11:26	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-ns5mk	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:11:26	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-ns5mk	Started	Started container kube-rbac-proxy
19:11:34 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:11:34 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:12:19 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:12:19 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:12:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	InferenceServiceReady	InferenceService [isvc-sklearn-v2-mixed] is Ready
19:12:22 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:12:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj	Killing	Stopping container kserve-container
19:12:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj	Killing	Stopping container kube-rbac-proxy
19:12:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj	Unhealthy	Readiness probe failed: Get "https://10.132.0.43:8643/healthz": dial tcp 10.132.0.43:8643: connect: connection refused
19:12:32 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-64bd78979b-m2tnj	Unhealthy	Readiness probe failed: dial tcp 10.132.0.43:8080: connect: connection refused
19:12:34	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-predictor-547dfc5669 from 0 to 1
19:12:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow": the object has been modified; please apply your changes to the latest version and try again
19:12:34	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-predictor-547dfc5669	SuccessfulCreate	Created pod: isvc-tensorflow-predictor-547dfc5669-stxg2
19:12:35	kserve-ci-e2e-test	multus	isvc-tensorflow-predictor-547dfc5669-stxg2	AddedInterface	Add eth0 [10.134.0.46/23] from ovn-kubernetes
19:12:35	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxg2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:12:35	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxg2	Created	Created container: storage-initializer
19:12:35 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-ns5mk	Unhealthy	Readiness probe failed: dial tcp 10.134.0.45:8080: connect: connection refused
19:12:35	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxg2	Started	Started container storage-initializer
19:12:40	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxg2	Pulling	Pulling image "tensorflow/serving:2.6.2"
19:12:43	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-ns5mk	Killing	Stopping container kserve-container
19:12:43	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-ns5mk	Killing	Stopping container kube-rbac-proxy
19:12:43	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-promote-predictor-674955ff8d from 2 to 1
19:12:43	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-674955ff8d	SuccessfulDelete	Deleted pod: isvc-canary-promote-predictor-674955ff8d-ns5mk
19:12:44	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxg2	Started	Started container kserve-container
19:12:44	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxg2	Created	Created container: kube-rbac-proxy
19:12:44	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxg2	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 3.854s (3.854s including waiting). Image size: 425873876 bytes.
19:12:44	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxg2	Created	Created container: kserve-container
19:12:44	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxg2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:12:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-ns5mk	Unhealthy	Readiness probe failed: Get "https://10.134.0.45:8643/healthz": dial tcp 10.134.0.45:8643: connect: connection refused
19:12:44	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxg2	Started	Started container kube-rbac-proxy
19:12:51 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxg2	Unhealthy	Readiness probe failed: dial tcp 10.134.0.46:8080: connect: connection refused
19:13:00	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-v2-predictor-85cf669bc9 from 1 to 2
19:13:00	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-njhv9	Killing	Stopping container kserve-container
19:13:00	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-njhv9	Killing	Stopping container kube-rbac-proxy
19:13:01	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-ktc84	Started	Started container storage-initializer
19:13:01 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:13:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	InferenceServiceReady	InferenceService [isvc-tensorflow] is Ready
19:13:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	InferenceServiceNotReady	InferenceService [isvc-canary-promote] is no longer Ready because of: PredictorReady, Ready, Stopped
19:13:01 (x22)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:13:01	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-ktc84	Created	Created container: storage-initializer
19:13:01	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-v2-predictor-85cf669bc9	SuccessfulCreate	Created pod: isvc-canary-promote-v2-predictor-85cf669bc9-ktc84
19:13:01	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-ktc84	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:13:01	kserve-ci-e2e-test	multus	isvc-canary-promote-v2-predictor-85cf669bc9-ktc84	AddedInterface	Add eth0 [10.132.0.45/23] from ovn-kubernetes
19:13:02 (x11)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-njhv9	Unhealthy	Readiness probe failed: dial tcp 10.132.0.44:8080: connect: connection refused
19:13:02	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-674955ff8d-njhv9	Unhealthy	Readiness probe failed: Get "https://10.132.0.44:8643/healthz": dial tcp 10.132.0.44:8643: connect: connection refused
19:13:04 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:13:04 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:13:06	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-ktc84	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:13:06	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-ktc84	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" already present on machine
19:13:06	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-ktc84	Started	Started container kube-rbac-proxy
19:13:06	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-ktc84	Created	Created container: kube-rbac-proxy
19:13:06	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-ktc84	Started	Started container kserve-container
19:13:06	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-ktc84	Created	Created container: kserve-container
19:13:14	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxg2	Killing	Stopping container kube-rbac-proxy
19:13:14	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxg2	Killing	Stopping container kserve-container
19:13:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow-runtime": the object has been modified; please apply your changes to the latest version and try again
19:13:19	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-runtime-predictor-855cfcd677 from 0 to 1
19:13:19	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-runtime-predictor-855cfcd677	SuccessfulCreate	Created pod: isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s
19:13:20	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s	Created	Created container: storage-initializer
19:13:20	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s	Started	Started container storage-initializer
19:13:20	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:13:20	kserve-ci-e2e-test	multus	isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s	AddedInterface	Add eth0 [10.132.0.46/23] from ovn-kubernetes
19:13:26	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s	Pulling	Pulling image "tensorflow/serving:2.6.2"
19:13:30	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:13:30	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s	Created	Created container: kube-rbac-proxy
19:13:30	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s	Started	Started container kube-rbac-proxy
19:13:30	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s	Started	Started container kserve-container
19:13:30	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s	Created	Created container: kserve-container
19:13:30	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 3.827s (3.827s including waiting). Image size: 425873876 bytes.
19:13:37 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s	Unhealthy	Readiness probe failed: dial tcp 10.132.0.46:8080: connect: connection refused
19:13:41 (x6)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-stxg2	Unhealthy	Readiness probe failed: Get "https://10.134.0.46:8643/healthz": dial tcp 10.134.0.46:8643: connect: connection refused
19:13:47 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:13:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	InferenceServiceReady	InferenceService [isvc-tensorflow-runtime] is Ready
19:13:49 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:13:49 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:14:02	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s	Killing	Stopping container kserve-container
19:14:02	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s	Killing	Stopping container kube-rbac-proxy
19:14:07	kserve-ci-e2e-test	replicaset-controller	isvc-triton-predictor-5d7db58676	SuccessfulCreate	Created pod: isvc-triton-predictor-5d7db58676-xl8qh
19:14:07	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	UpdateFailed	Failed to update status for InferenceService "isvc-triton": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-triton": the object has been modified; please apply your changes to the latest version and try again
19:14:07	kserve-ci-e2e-test	deployment-controller	isvc-triton-predictor	ScalingReplicaSet	Scaled up replica set isvc-triton-predictor-5d7db58676 from 0 to 1
19:14:08	kserve-ci-e2e-test	multus	isvc-triton-predictor-5d7db58676-xl8qh	AddedInterface	Add eth0 [10.134.0.47/23] from ovn-kubernetes
19:14:08	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-xl8qh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:14:08	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-xl8qh	Created	Created container: storage-initializer
19:14:08	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-xl8qh	Started	Started container storage-initializer
19:14:13	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-xl8qh	Pulling	Pulling image "nvcr.io/nvidia/tritonserver:23.05-py3"
19:14:32 (x6)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-vzv7s	Unhealthy	Readiness probe failed: Get "https://10.132.0.46:8643/healthz": dial tcp 10.132.0.46:8643: connect: connection refused
19:14:33 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	InferenceServiceReady	InferenceService [isvc-canary-promote] is Ready
19:14:41	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-ktc84	Killing	Stopping container kserve-container
19:14:41	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-ktc84	Killing	Stopping container kube-rbac-proxy
19:14:41	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw	Killing	Stopping container kserve-container
19:14:41	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw	Killing	Stopping container kube-rbac-proxy
19:14:43 (x11)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-ktc84	Unhealthy	Readiness probe failed: dial tcp 10.132.0.45:8080: connect: connection refused
19:14:43	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-ktc84	Unhealthy	Readiness probe failed: Get "https://10.132.0.45:8643/healthz": dial tcp 10.132.0.45:8643: connect: connection refused
19:14:43 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw	Unhealthy	Readiness probe failed: dial tcp 10.134.0.44:8080: connect: connection refused
19:14:43	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw	Unhealthy	Readiness probe failed: Get "https://10.134.0.44:8643/healthz": dial tcp 10.134.0.44:8643: connect: connection refused
19:14:46	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b	Started	Started container storage-initializer
19:14:46	kserve-ci-e2e-test	multus	isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b	AddedInterface	Add eth0 [10.132.0.47/23] from ovn-kubernetes
19:14:46	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-predictor-serving-cert" not found
19:14:46	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-6bd4b7c9bd	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt
19:14:46	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:14:46	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b	Created	Created container: storage-initializer
19:14:46	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-predictor-serving-cert" not found
19:14:46	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-predictor-6bd4b7c9bd from 0 to 2
19:14:46	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-6bd4b7c9bd	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t
19:14:46	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-v2-predictor-7b857bf96f from 0 to 1
19:14:46	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-v2-predictor-7b857bf96f	SuccessfulCreate	Created pod: isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b
19:14:47	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:14:47	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt	AddedInterface	Add eth0 [10.132.0.48/23] from ovn-kubernetes
19:14:47	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t	AddedInterface	Add eth0 [10.134.0.48/23] from ovn-kubernetes
19:14:47	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:14:47	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t	Created	Created container: storage-initializer
19:14:47	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t	Started	Started container storage-initializer
19:14:47	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt	Started	Started container storage-initializer
19:14:47	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt	Created	Created container: storage-initializer
19:14:51	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt	Created	Created container: kserve-container
19:14:51	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" already present on machine
19:14:51	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" already present on machine
19:14:51	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b	Created	Created container: kserve-container
19:14:51	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b	Started	Started container kserve-container
19:14:51	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:14:51	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b	Created	Created container: kube-rbac-proxy
19:14:51	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b	Started	Started container kube-rbac-proxy
19:14:51	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt	Started	Started container kserve-container
19:14:51	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:14:51	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt	Created	Created container: kube-rbac-proxy
19:14:51	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt	Started	Started container kube-rbac-proxy
19:14:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t	Created	Created container: kserve-container
19:14:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" already present on machine
19:14:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t	Started	Started container kserve-container
19:14:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:14:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t	Created	Created container: kube-rbac-proxy
19:14:52	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t	Started	Started container kube-rbac-proxy
19:16:01 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt	Unhealthy	Readiness probe failed: dial tcp 10.132.0.48:8080: connect: connection refused
19:16:08	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-xl8qh	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:16:08	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-xl8qh	Pulled	Successfully pulled image "nvcr.io/nvidia/tritonserver:23.05-py3" in 1m55.303s (1m55.303s including waiting). Image size: 12907074623 bytes.
19:16:08	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-xl8qh	Started	Started container kube-rbac-proxy
19:16:08	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt	Killing	Stopping container kserve-container
19:16:08	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt	Killing	Stopping container kube-rbac-proxy
19:16:08 (x3)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	UpdateFailed	Failed to update status for InferenceService "isvc-canary-rollback": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again
19:16:08 (x3)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again
19:16:08	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-rollback-predictor-6bd4b7c9bd from 2 to 1
19:16:08	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-xl8qh	Created	Created container: kube-rbac-proxy
19:16:08	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-6bd4b7c9bd	SuccessfulDelete	Deleted pod: isvc-canary-rollback-predictor-6bd4b7c9bd-sskwt
19:16:08	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-xl8qh	Created	Created container: kserve-container
19:16:08	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-xl8qh	Started	Started container kserve-container
19:16:14 (x3)	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-xl8qh	Unhealthy	Readiness probe failed: dial tcp 10.134.0.47:8080: connect: connection refused
19:16:15 (x11)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t	Unhealthy	Readiness probe failed: dial tcp 10.134.0.48:8080: connect: connection refused
19:16:22 (x9)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:16:22 (x9)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:16:24	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	InferenceServiceReady	InferenceService [isvc-triton] is Ready
19:16:24 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:16:25 (x19)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:16:26	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-6bd4b7c9bd	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw
19:16:26	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-predictor-6bd4b7c9bd from 1 to 2
19:16:26	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b	Killing	Stopping container kube-rbac-proxy
19:16:26	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b	Killing	Stopping container kserve-container
19:16:27	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw	Started	Started container storage-initializer
19:16:27	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw	AddedInterface	Add eth0 [10.132.0.49/23] from ovn-kubernetes
19:16:27	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:16:27	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw	Created	Created container: storage-initializer
19:16:28	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b	Unhealthy	Readiness probe failed: Get "https://10.132.0.47:8643/healthz": dial tcp 10.132.0.47:8643: connect: connection refused
19:16:28 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b857bf96f-n9p2b	Unhealthy	Readiness probe failed: dial tcp 10.132.0.47:8080: connect: connection refused
19:16:30	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-xl8qh	Killing	Stopping container kube-rbac-proxy
19:16:30	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-xl8qh	Killing	Stopping container kserve-container
19:16:31	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw	Started	Started container kube-rbac-proxy
19:16:31	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw	Created	Created container: kube-rbac-proxy
19:16:31	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:16:31	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw	Started	Started container kserve-container
19:16:31	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw	Created	Created container: kserve-container
19:16:31	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" already present on machine
19:16:35	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-km76b	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-predictor-serving-cert" not found
19:16:35	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-predictor-7566d65bf7	SuccessfulCreate	Created pod: isvc-xgboost-predictor-7566d65bf7-km76b
19:16:35	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-predictor-7566d65bf7 from 0 to 1
19:16:35 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost": the object has been modified; please apply your changes to the latest version and try again
19:16:35 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost": the object has been modified; please apply your changes to the latest version and try again
19:16:36	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-km76b	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:16:36	kserve-ci-e2e-test	multus	isvc-xgboost-predictor-7566d65bf7-km76b	AddedInterface	Add eth0 [10.134.0.49/23] from ovn-kubernetes
19:16:36	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-km76b	Started	Started container storage-initializer
19:16:36	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-km76b	Created	Created container: storage-initializer
19:16:40	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-km76b	Pulling	Pulling image "kserve/xgbserver:latest"
19:17:00	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-km76b	Created	Created container: kserve-container
19:17:00	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-km76b	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 20.031s (20.031s including waiting). Image size: 1332365570 bytes.
19:17:01	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-km76b	Started	Started container kube-rbac-proxy
19:17:01	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-km76b	Started	Started container kserve-container
19:17:01	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-km76b	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:17:01	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-km76b	Created	Created container: kube-rbac-proxy
19:17:20 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:17:20 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:17:39 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	InferenceServiceReady	InferenceService [isvc-canary-rollback] is Ready
19:17:46	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t	Killing	Stopping container kube-rbac-proxy
19:17:46	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw	Killing	Stopping container kube-rbac-proxy
19:17:46	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw	Killing	Stopping container kserve-container
19:17:46	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t	Killing	Stopping container kserve-container
19:17:49 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw	Unhealthy	Readiness probe failed: dial tcp 10.132.0.49:8080: connect: connection refused
19:17:49	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t	Unhealthy	Readiness probe failed: Get "https://10.134.0.48:8643/healthz": dial tcp 10.134.0.48:8643: connect: connection refused
19:17:49	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6bd4b7c9bd-k9mbw	Unhealthy	Readiness probe failed: Get "https://10.132.0.49:8643/healthz": dial tcp 10.132.0.49:8643: connect: connection refused
19:17:51	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-69f76fb68d	SuccessfulCreate	Created pod: isvc-canary-stop-predictor-69f76fb68d-4dh7t
19:17:51	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-69f76fb68d	SuccessfulCreate	Created pod: isvc-canary-stop-predictor-69f76fb68d-zjjqs
19:17:51 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again
19:17:51 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	UpdateFailed	Failed to update status for InferenceService "isvc-canary-stop": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again
19:17:51	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-stop-predictor-69f76fb68d from 0 to 2
19:17:51	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-stop-v2-predictor-5b684d6bb9 from 0 to 1
19:17:51	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-zjjqs	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-stop-predictor-serving-cert" not found
19:17:51	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-v2-predictor-5b684d6bb9	SuccessfulCreate	Created pod: isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd
19:17:52	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:17:52	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd	Created	Created container: storage-initializer
19:17:52	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-4dh7t	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-stop-predictor-serving-cert" not found
19:17:52	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd	Started	Started container storage-initializer
19:17:52	kserve-ci-e2e-test	multus	isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd	AddedInterface	Add eth0 [10.134.0.50/23] from ovn-kubernetes
19:17:52	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-zjjqs	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:17:52	kserve-ci-e2e-test	multus	isvc-canary-stop-predictor-69f76fb68d-zjjqs	AddedInterface	Add eth0 [10.132.0.50/23] from ovn-kubernetes
19:17:53	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-4dh7t	Started	Started container storage-initializer
19:17:53	kserve-ci-e2e-test	multus	isvc-canary-stop-predictor-69f76fb68d-4dh7t	AddedInterface	Add eth0 [10.134.0.51/23] from ovn-kubernetes
19:17:53	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-4dh7t	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:17:53	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-4dh7t	Created	Created container: storage-initializer
19:17:53	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-zjjqs	Started	Started container storage-initializer
19:17:53	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-zjjqs	Created	Created container: storage-initializer
19:17:57	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" already present on machine
19:17:57	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-zjjqs	Started	Started container kube-rbac-proxy
19:17:57	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-4dh7t	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" already present on machine
19:17:57	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-4dh7t	Started	Started container kserve-container
19:17:57	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd	Created	Created container: kserve-container
19:17:57	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-4dh7t	Started	Started container kube-rbac-proxy
19:17:57	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-zjjqs	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" already present on machine
19:17:57	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-zjjqs	Created	Created container: kserve-container
19:17:57	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-zjjqs	Started	Started container kserve-container
19:17:57	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-zjjqs	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:17:57	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-zjjqs	Created	Created container: kube-rbac-proxy
19:17:57	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd	Started	Started container kserve-container
19:17:57	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-4dh7t	Created	Created container: kserve-container
19:17:57	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd	Started	Started container kube-rbac-proxy
19:17:57	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-4dh7t	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:17:57	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:17:57	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd	Created	Created container: kube-rbac-proxy
19:17:57	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-4dh7t	Created	Created container: kube-rbac-proxy
19:18:05 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:18:05 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:18:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	InferenceServiceReady	InferenceService [isvc-xgboost] is Ready
19:18:08 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:18:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-km76b	Killing	Stopping container kube-rbac-proxy
19:18:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-km76b	Killing	Stopping container kserve-container
19:18:18	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-km76b	Unhealthy	Readiness probe failed: Get "https://10.134.0.49:8643/healthz": dial tcp 10.134.0.49:8643: connect: connection refused
19:18:18 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-km76b	Unhealthy	Readiness probe failed: dial tcp 10.134.0.49:8080: connect: connection refused
19:18:20	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
19:18:20	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-mlserver-predictor-7b55b84f54	SuccessfulCreate	Created pod: isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d
19:18:20	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-mlserver-predictor-7b55b84f54 from 0 to 1
19:18:20 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
19:18:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
19:18:21	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d	Started	Started container storage-initializer
19:18:21	kserve-ci-e2e-test	multus	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d	AddedInterface	Add eth0 [10.132.0.51/23] from ovn-kubernetes
19:18:21	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:18:21	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d	Created	Created container: storage-initializer
19:18:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d	Started	Started container kserve-container
19:18:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:18:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d	Created	Created container: kube-rbac-proxy
19:18:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d	Started	Started container kube-rbac-proxy
19:18:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d	Created	Created container: kserve-container
19:18:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:19:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	InferenceServiceReady	InferenceService [isvc-xgboost-v2-mlserver] is Ready
19:19:04 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:19:05 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:19:05 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:19:05 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-4dh7t	Unhealthy	Readiness probe failed: dial tcp 10.134.0.51:8080: connect: connection refused
19:19:10	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d	Killing	Stopping container kube-rbac-proxy
19:19:10	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d	Killing	Stopping container kserve-container
19:19:14	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d	Unhealthy	Readiness probe failed: Get "http://10.132.0.51:8080/v2/models/isvc-xgboost-v2-mlserver/ready": dial tcp 10.132.0.51:8080: connect: connection refused
19:19:14	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-cw69d	Unhealthy	Readiness probe failed: Get "https://10.132.0.51:8643/healthz": dial tcp 10.132.0.51:8643: connect: connection refused
19:19:15	kserve-ci-e2e-test	replicaset-controller	xgboost-v2-mlserver-predictor-59944bcf4c	SuccessfulCreate	Created pod: xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv
19:19:15	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-4dh7t	Killing	Stopping container kserve-container
19:19:15	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-4dh7t	Killing	Stopping container kube-rbac-proxy
19:19:15	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
19:19:15	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
19:19:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	InferenceServiceReady	InferenceService [isvc-canary-stop] is Ready
19:19:15 (x21)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:19:15	kserve-ci-e2e-test	deployment-controller	xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set xgboost-v2-mlserver-predictor-59944bcf4c from 0 to 1
19:19:15	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-69f76fb68d	SuccessfulDelete	Deleted pod: isvc-canary-stop-predictor-69f76fb68d-4dh7t
19:19:15	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-stop-predictor-69f76fb68d from 2 to 1
19:19:16	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv	Started	Started container storage-initializer
19:19:16	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv	Created	Created container: storage-initializer
19:19:16	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:19:16	kserve-ci-e2e-test	multus	xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv	AddedInterface	Add eth0 [10.132.0.52/23] from ovn-kubernetes
19:19:20	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:19:20	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv	Started	Started container kserve-container
19:19:20	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv	Created	Created container: kserve-container
19:19:20	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:19:21	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-zjjqs	Killing	Stopping container kube-rbac-proxy
19:19:21	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-zjjqs	Killing	Stopping container kserve-container
19:19:21	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd	Killing	Stopping container kserve-container
19:19:21	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv	Created	Created container: kube-rbac-proxy
19:19:21	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd	Killing	Stopping container kube-rbac-proxy
19:19:21	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv	Started	Started container kube-rbac-proxy
19:19:24	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-zjjqs	Unhealthy	Readiness probe failed: Get "https://10.132.0.50:8643/healthz": dial tcp 10.132.0.50:8643: connect: connection refused
19:19:24 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-69f76fb68d-zjjqs	Unhealthy	Readiness probe failed: dial tcp 10.132.0.50:8080: connect: connection refused
19:19:25 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd	Unhealthy	Readiness probe failed: dial tcp 10.134.0.50:8080: connect: connection refused
19:19:25	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd	Unhealthy	Readiness probe failed: Get "https://10.134.0.50:8643/healthz": dial tcp 10.134.0.50:8643: connect: connection refused
19:19:31	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-predictor-54f85ddf88 from 0 to 1
19:19:32 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm": the object has been modified; please apply your changes to the latest version and try again
19:19:32	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm": the object has been modified; please apply your changes to the latest version and try again
19:19:32	kserve-ci-e2e-test	multus	isvc-lightgbm-predictor-54f85ddf88-574r7	AddedInterface	Add eth0 [10.134.0.52/23] from ovn-kubernetes
19:19:32	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-predictor-54f85ddf88	SuccessfulCreate	Created pod: isvc-lightgbm-predictor-54f85ddf88-574r7
19:19:32	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-574r7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:19:32	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-574r7	Created	Created container: storage-initializer
19:19:32	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-574r7	Started	Started container storage-initializer
19:19:36	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-574r7	Pulling	Pulling image "kserve/lgbserver:latest"
19:19:43	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-574r7	Pulled	Successfully pulled image "kserve/lgbserver:latest" in 6.285s (6.285s including waiting). Image size: 632192787 bytes.
19:19:43	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-574r7	Created	Created container: kserve-container
19:19:43	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-574r7	Started	Started container kserve-container
19:19:43	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-574r7	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:19:43	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-574r7	Created	Created container: kube-rbac-proxy
19:19:43	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-574r7	Started	Started container kube-rbac-proxy
19:19:45 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:19:45 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:19:46	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:19:46	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:19:59 (x12)	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:19:59	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	InferenceServiceReady	InferenceService [xgboost-v2-mlserver] is Ready
19:20:00	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:20:00	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:20:05	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv	Killing	Stopping container kube-rbac-proxy
19:20:05	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv	Killing	Stopping container kserve-container
19:20:08	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv	Unhealthy	Readiness probe failed: Get "http://10.132.0.52:8080/v2/models/xgboost-v2-mlserver/ready": dial tcp 10.132.0.52:8080: connect: connection refused
19:20:08	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-dx7lv	Unhealthy	Readiness probe failed: Get "https://10.132.0.52:8643/healthz": dial tcp 10.132.0.52:8643: connect: connection refused
19:20:11	kserve-ci-e2e-test	multus	isvc-xgboost-runtime-predictor-54dd488c85-w4qb6	AddedInterface	Add eth0 [10.134.0.53/23] from ovn-kubernetes
19:20:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-runtime": the object has been modified; please apply your changes to the latest version and try again
19:20:11	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-runtime-predictor-54dd488c85	SuccessfulCreate	Created pod: isvc-xgboost-runtime-predictor-54dd488c85-w4qb6
19:20:11	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-runtime-predictor-54dd488c85 from 0 to 1
19:20:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-w4qb6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:20:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-w4qb6	Created	Created container: storage-initializer
19:20:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-w4qb6	Started	Started container storage-initializer
19:20:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-w4qb6	Pulled	Container image "kserve/xgbserver:latest" already present on machine
19:20:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-w4qb6	Created	Created container: kube-rbac-proxy
19:20:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-w4qb6	Created	Created container: kserve-container
19:20:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-w4qb6	Started	Started container kserve-container
19:20:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-w4qb6	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:20:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-w4qb6	Started	Started container kube-rbac-proxy
19:20:50 (x9)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-574r7	Unhealthy	Readiness probe failed: dial tcp 10.134.0.52:8080: connect: connection refused
19:20:56 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:20:56 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:21:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	InferenceServiceReady	InferenceService [isvc-lightgbm] is Ready
19:21:01 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:21:02	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-574r7	Killing	Stopping container kserve-container
19:21:02 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:21:02 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:21:02	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-574r7	Killing	Stopping container kube-rbac-proxy
19:21:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-574r7	Unhealthy	Readiness probe failed: Get "https://10.134.0.52:8643/healthz": dial tcp 10.134.0.52:8643: connect: connection refused
19:21:07	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-runtime": the object has been modified; please apply your changes to the latest version and try again
19:21:07	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
19:21:07	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-runtime-predictor-577f7db7cb	SuccessfulCreate	Created pod: isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k
19:21:07	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-runtime-predictor-577f7db7cb from 0 to 1
19:21:08	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:21:08	kserve-ci-e2e-test	multus	isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k	AddedInterface	Add eth0 [10.132.0.53/23] from ovn-kubernetes
19:21:08	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k	Started	Started container storage-initializer
19:21:08	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k	Created	Created container: storage-initializer
19:21:13	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k	Pulling	Pulling image "kserve/lgbserver:latest"
19:21:19	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k	Started	Started container kube-rbac-proxy
19:21:19	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k	Created	Created container: kube-rbac-proxy
19:21:19	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:21:19	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k	Started	Started container kserve-container
19:21:19	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k	Created	Created container: kserve-container
19:21:19	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k	Pulled	Successfully pulled image "kserve/lgbserver:latest" in 6.439s (6.439s including waiting). Image size: 632192787 bytes.
19:21:23	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-runtime] is Ready
19:21:23 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:21:26 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:21:26 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:21:31	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-w4qb6	Killing	Stopping container kserve-container
19:21:31	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-w4qb6	Killing	Stopping container kube-rbac-proxy
19:21:32 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-w4qb6	Unhealthy	Readiness probe failed: dial tcp 10.134.0.53:8080: connect: connection refused
19:21:32	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-w4qb6	Unhealthy	Readiness probe failed: Get "https://10.134.0.53:8643/healthz": dial tcp 10.134.0.53:8643: connect: connection refused
19:21:36	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
19:21:36 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
19:21:36	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-runtime-predictor-7787c55dc9	SuccessfulCreate	Created pod: isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs
19:21:36	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-runtime-predictor-7787c55dc9 from 0 to 1
19:21:36	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
19:21:37	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:21:37	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs	Created	Created container: storage-initializer
19:21:37	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs	Started	Started container storage-initializer
19:21:37	kserve-ci-e2e-test	multus	isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs	AddedInterface	Add eth0 [10.132.0.54/23] from ovn-kubernetes
19:21:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs	Created	Created container: kube-rbac-proxy
19:21:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:21:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs	Started	Started container kube-rbac-proxy
19:21:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs	Created	Created container: kserve-container
19:21:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs	Started	Started container kserve-container
19:21:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:21:52 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:21:52 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:22:19	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
19:22:21 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:22:21 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:22:27 (x9)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k	Unhealthy	Readiness probe failed: dial tcp 10.132.0.53:8080: connect: connection refused
19:22:29 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:22:29	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-v2-runtime] is Ready
19:22:36	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs	Killing	Stopping container kserve-container
19:22:36	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:22:36	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs	Killing	Stopping container kube-rbac-proxy
19:22:36	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:22:37 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:22:37	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-runtime] is Ready
19:22:37 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:22:37 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:22:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs	Unhealthy	Readiness probe failed: Get "https://10.132.0.54:8643/healthz": dial tcp 10.132.0.54:8643: connect: connection refused
19:22:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-82bqs	Unhealthy	Readiness probe failed: Get "http://10.132.0.54:8080/v2/models/isvc-xgboost-v2-runtime/ready": dial tcp 10.132.0.54:8080: connect: connection refused
19:22:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
19:22:41 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
19:22:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-bd4kg	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-v2-predictor-serving-cert" not found
19:22:41	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-predictor-fcc668bc from 0 to 1
19:22:41	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-predictor-fcc668bc	SuccessfulCreate	Created pod: isvc-xgboost-v2-predictor-fcc668bc-bd4kg
19:22:42	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-bd4kg	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:22:42	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-bd4kg	Started	Started container storage-initializer
19:22:42	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-bd4kg	Created	Created container: storage-initializer
19:22:42	kserve-ci-e2e-test	multus	isvc-xgboost-v2-predictor-fcc668bc-bd4kg	AddedInterface	Add eth0 [10.132.0.55/23] from ovn-kubernetes
19:22:46	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-bd4kg	Pulling	Pulling image "kserve/xgbserver:latest"
19:22:48	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k	Killing	Stopping container kserve-container
19:22:48	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k	Killing	Stopping container kube-rbac-proxy
19:22:52	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-8xb6k	Unhealthy	Readiness probe failed: Get "https://10.132.0.53:8643/healthz": dial tcp 10.132.0.53:8643: connect: connection refused
19:22:53	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5	SuccessfulCreate	Created pod: isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6
19:22:53	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-runtime-predictor-7f4957c5d5 from 0 to 1
19:22:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
19:22:54	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
19:22:55	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6	AddedInterface	Add eth0 [10.132.0.56/23] from ovn-kubernetes
19:22:55	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:22:55	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6	Created	Created container: storage-initializer
19:22:55	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6	Started	Started container storage-initializer
19:23:03	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:23:03	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-bd4kg	Started	Started container kube-rbac-proxy
19:23:03	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-bd4kg	Created	Created container: kube-rbac-proxy
19:23:03	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-bd4kg	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:23:03	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-bd4kg	Started	Started container kserve-container
19:23:03	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-bd4kg	Created	Created container: kserve-container
19:23:03	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-bd4kg	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 16.887s (16.887s including waiting). Image size: 1332365570 bytes.
19:23:04	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6	Started	Started container kserve-container
19:23:04	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6	Created	Created container: kserve-container
19:23:04	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:23:04	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6	Created	Created container: kube-rbac-proxy
19:23:04	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6	Started	Started container kube-rbac-proxy
19:23:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:23:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:23:41 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:23:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-runtime] is Ready
19:23:42 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:23:42 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:23:43	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6	Killing	Stopping container kserve-container
19:23:43	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-lg8p6	Killing	Stopping container kube-rbac-proxy
19:23:49	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8	Started	Started container storage-initializer
19:23:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
19:23:49	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5	SuccessfulCreate	Created pod: isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8
19:23:49	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5 from 0 to 1
19:23:49	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:23:49	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8	AddedInterface	Add eth0 [10.132.0.57/23] from ovn-kubernetes
19:23:49	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8	Created	Created container: storage-initializer
19:23:54	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8	Started	Started container kserve-container
19:23:54	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8	Started	Started container kube-rbac-proxy
19:23:54	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8	Pulled	Container image "kserve/lgbserver:latest" already present on machine
19:23:54	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8	Created	Created container: kserve-container
19:23:54	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:23:54	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8	Created	Created container: kube-rbac-proxy
19:23:57	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:23:57	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:24:00 (x8)	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-bd4kg	Unhealthy	Readiness probe failed: dial tcp 10.132.0.55:8080: connect: connection refused
19:24:01 (x3)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8	Unhealthy	Readiness probe failed: dial tcp 10.132.0.57:8080: connect: connection refused
19:24:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-bd4kg	Killing	Stopping container kube-rbac-proxy
19:24:11 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:24:11 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:24:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	InferenceServiceReady	InferenceService [isvc-xgboost-v2] is Ready
19:24:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-kserve] is Ready
19:24:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-bd4kg	Killing	Stopping container kserve-container
19:24:16	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-predictor-5584f7c89b from 0 to 1
19:24:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3": the object has been modified; please apply your changes to the latest version and try again
19:24:16	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-predictor-5584f7c89b	SuccessfulCreate	Created pod: isvc-sklearn-s3-predictor-5584f7c89b-l4jlw
19:24:17	kserve-ci-e2e-test	multus	isvc-sklearn-s3-predictor-5584f7c89b-l4jlw	AddedInterface	Add eth0 [10.132.0.58/23] from ovn-kubernetes
19:24:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-5584f7c89b-l4jlw	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:24:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-5584f7c89b-l4jlw	Created	Created container: storage-initializer
19:24:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-5584f7c89b-l4jlw	Started	Started container storage-initializer
19:24:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-5584f7c89b-l4jlw	Created	Created container: kube-rbac-proxy
19:24:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-5584f7c89b-l4jlw	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:24:19 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:24:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-5584f7c89b-l4jlw	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" already present on machine
19:24:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-5584f7c89b-l4jlw	Created	Created container: kserve-container
19:24:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-5584f7c89b-l4jlw	Started	Started container kserve-container
19:24:19 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:24:19	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8	Killing	Stopping container kserve-container
19:24:19	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6p5k8	Killing	Stopping container kube-rbac-proxy
19:24:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-5584f7c89b-l4jlw	Started	Started container kube-rbac-proxy
19:24:24	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
19:24:24	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-mlflow-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-mlflow-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
19:24:24	kserve-ci-e2e-test	deployment-controller	isvc-mlflow-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-mlflow-v2-runtime-predictor-6456f94494 from 0 to 1
19:24:24	kserve-ci-e2e-test	replicaset-controller	isvc-mlflow-v2-runtime-predictor-6456f94494	SuccessfulCreate	Created pod: isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg
19:24:25	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:24:25	kserve-ci-e2e-test	multus	isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg	AddedInterface	Add eth0 [10.132.0.59/23] from ovn-kubernetes
19:24:25	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg	Started	Started container storage-initializer
19:24:25	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg	Created	Created container: storage-initializer
19:24:30	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:24:30	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg	Created	Created container: kube-rbac-proxy
19:24:30	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg	Started	Started container kube-rbac-proxy
19:24:30	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg	Created	Created container: kserve-container
19:24:30	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:24:30	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg	Started	Started container kserve-container
19:24:46 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:24:46 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:25:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	InferenceServiceReady	InferenceService [isvc-mlflow-v2-runtime] is Ready
19:25:08 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:25:09 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:25:09 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:25:14	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg	Killing	Stopping container kube-rbac-proxy
19:25:14	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-w7crg	Killing	Stopping container kserve-container
19:25:19 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-mcp": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-mcp": the object has been modified; please apply your changes to the latest version and try again
19:25:19	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-mcp-predictor-6d944dbd5f	SuccessfulCreate	Created pod: isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n
19:25:19	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-mcp-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-mcp-predictor-6d944dbd5f from 0 to 1
19:25:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-mcp": the object has been modified; please apply your changes to the latest version and try again
19:25:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n	Created	Created container: storage-initializer
19:25:20	kserve-ci-e2e-test	multus	isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n	AddedInterface	Add eth0 [10.134.0.54/23] from ovn-kubernetes
19:25:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n	Started	Started container storage-initializer
19:25:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:25:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" already present on machine
19:25:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n	Created	Created container: kserve-container
19:25:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n	Started	Started container kserve-container
19:25:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n	Pulling	Pulling image "quay.io/opendatahub/kserve-agent:latest"
19:25:26 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-5584f7c89b-l4jlw	Unhealthy	Readiness probe failed: dial tcp 10.132.0.58:8080: connect: connection refused
19:25:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n	Created	Created container: kube-rbac-proxy
19:25:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n	Started	Started container kserve-agent
19:25:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n	Started	Started container kube-rbac-proxy
19:25:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent:latest" in 2.709s (2.709s including waiting). Image size: 239481897 bytes.
19:25:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n	Created	Created container: kserve-agent
19:25:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:25:31 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:25:31 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:25:36 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:25:36	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	InferenceServiceReady	InferenceService [isvc-sklearn-s3] is Ready
19:25:37	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6
19:25:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-5584f7c89b-l4jlw	Killing	Stopping container kube-rbac-proxy
19:25:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-5584f7c89b-l4jlw	Killing	Stopping container kserve-container
19:25:37	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6 from 0 to 1
19:25:37	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-pass": the object has been modified; please apply your changes to the latest version and try again
19:25:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6	Started	Started container storage-initializer
19:25:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6	Created	Created container: storage-initializer
19:25:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:25:37	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6	AddedInterface	Add eth0 [10.132.0.60/23] from ovn-kubernetes
19:25:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" already present on machine
19:25:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:25:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6	Created	Created container: kserve-container
19:25:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6	Created	Created container: kube-rbac-proxy
19:25:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6	Started	Started container kserve-container
19:25:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6	Started	Started container kube-rbac-proxy
19:25:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-5584f7c89b-l4jlw	Unhealthy	Readiness probe failed: Get "https://10.132.0.58:8643/healthz": dial tcp 10.132.0.58:8643: connect: connection refused
19:26:22 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:26:22 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:26:34 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:26:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	InferenceServiceReady	InferenceService [isvc-sklearn-mcp] is Ready
19:26:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n	Killing	Stopping container kserve-agent
19:26:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n	Killing	Stopping container kube-rbac-proxy
19:26:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n	Killing	Stopping container kserve-container
19:26:44	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-predictor-6d5c9fd4f6	SuccessfulCreate	Created pod: isvc-paddle-predictor-6d5c9fd4f6-hmhx6
19:26:44	kserve-ci-e2e-test	deployment-controller	isvc-paddle-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-predictor-6d5c9fd4f6 from 0 to 1
19:26:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	UpdateFailed	Failed to update status for InferenceService "isvc-paddle": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle": the object has been modified; please apply your changes to the latest version and try again
19:26:45	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-hmhx6	Created	Created container: storage-initializer
19:26:45	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-hmhx6	Started	Started container storage-initializer
19:26:45	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-hmhx6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:26:45	kserve-ci-e2e-test	multus	isvc-paddle-predictor-6d5c9fd4f6-hmhx6	AddedInterface	Add eth0 [10.132.0.61/23] from ovn-kubernetes
19:26:50	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-hmhx6	Pulling	Pulling image "kserve/paddleserver:latest"
19:26:52 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:26:52 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:26:57	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-global-pass] is Ready
19:26:57 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:27:01	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-hmhx6	Created	Created container: kserve-container
19:27:01	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-hmhx6	Pulled	Successfully pulled image "kserve/paddleserver:latest" in 11.803s (11.803s including waiting). Image size: 1175702273 bytes.
19:27:01	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-hmhx6	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:27:01	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-hmhx6	Started	Started container kserve-container
19:27:02	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-hmhx6	Started	Started container kube-rbac-proxy
19:27:02	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-hmhx6	Created	Created container: kube-rbac-proxy
19:27:04 (x4)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n	Unhealthy	Readiness probe failed: Get "http://10.134.0.54:8080/v1/models/isvc-sklearn-mcp": dial tcp 10.134.0.54:8080: connect: connection refused
19:27:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6	Killing	Stopping container kserve-container
19:27:07 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6	Unhealthy	Readiness probe failed: dial tcp 10.132.0.60:8080: connect: connection refused
19:27:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6	Killing	Stopping container kube-rbac-proxy
19:27:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-758ccfc6-l6kg6	Unhealthy	Readiness probe failed: Get "https://10.132.0.60:8643/healthz": dial tcp 10.132.0.60:8643: connect: connection refused
19:27:08	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k
19:27:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-fail": the object has been modified; please apply your changes to the latest version and try again
19:27:08	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k	AddedInterface	Add eth0 [10.134.0.55/23] from ovn-kubernetes
19:27:08	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6 from 0 to 1
19:27:09 (x6)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n	Unhealthy	Readiness probe failed: Get "https://10.134.0.54:8643/healthz": dial tcp 10.134.0.54:8643: connect: connection refused
19:27:13 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:27:13 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k	Started	Started container storage-initializer
19:27:13 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k	Created	Created container: storage-initializer
19:27:16 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:27:16	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k	BackOff	Back-off restarting failed container storage-initializer in pod isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k_kserve-ci-e2e-test(af4989c5-191b-49ce-8a00-0896badabb0c)
19:27:19	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6
19:27:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-pass": the object has been modified; please apply your changes to the latest version and try again
19:27:19	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b from 0 to 1
19:27:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6	Created	Created container: storage-initializer
19:27:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6	Started	Started container storage-initializer
19:27:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:27:20	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6	AddedInterface	Add eth0 [10.134.0.56/23] from ovn-kubernetes
19:27:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6	Started	Started container kserve-container
19:27:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:27:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6	Created	Created container: kube-rbac-proxy
19:27:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6	Started	Started container kube-rbac-proxy
19:27:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6	Created	Created container: kserve-container
19:27:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" already present on machine
19:27:29 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:27:29 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:27:44	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:27:44	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:27:48 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:27:48	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	InferenceServiceReady	InferenceService [isvc-paddle] is Ready
19:27:49 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:27:49 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:27:56	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-hmhx6	Killing	Stopping container kube-rbac-proxy
19:27:56	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-hmhx6	Killing	Stopping container kserve-container
19:27:58	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-hmhx6	Unhealthy	Readiness probe failed: Get "https://10.132.0.61:8643/healthz": dial tcp 10.132.0.61:8643: connect: connection refused
19:27:58 (x7)	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-hmhx6	Unhealthy	Readiness probe failed: dial tcp 10.132.0.61:8080: connect: connection refused
19:28:01	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-ffwrv	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-paddle-runtime-predictor-serving-cert" not found
19:28:01	kserve-ci-e2e-test	deployment-controller	isvc-paddle-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-runtime-predictor-58b99fff76 from 0 to 1
19:28:01	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-runtime-predictor-58b99fff76	SuccessfulCreate	Created pod: isvc-paddle-runtime-predictor-58b99fff76-ffwrv
19:28:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-runtime": the object has been modified; please apply your changes to the latest version and try again
19:28:02	kserve-ci-e2e-test	multus	isvc-paddle-runtime-predictor-58b99fff76-ffwrv	AddedInterface	Add eth0 [10.132.0.62/23] from ovn-kubernetes
19:28:02	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-ffwrv	Started	Started container storage-initializer
19:28:02	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-ffwrv	Created	Created container: storage-initializer
19:28:02	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-ffwrv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:28:07	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-ffwrv	Created	Created container: kube-rbac-proxy
19:28:07	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-ffwrv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:28:07	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-ffwrv	Started	Started container kserve-container
19:28:07	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-ffwrv	Created	Created container: kserve-container
19:28:07	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-ffwrv	Pulled	Container image "kserve/paddleserver:latest" already present on machine
19:28:07	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-ffwrv	Started	Started container kube-rbac-proxy
19:28:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:28:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:28:34 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:28:34 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:28:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-custom-pass] is Ready
19:28:39 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:28:49 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6	Unhealthy	Readiness probe failed: dial tcp 10.134.0.56:8080: connect: connection refused
19:28:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6	Killing	Stopping container kserve-container
19:28:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6	Killing	Stopping container kube-rbac-proxy
19:28:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6	Unhealthy	Readiness probe failed: Get "https://10.134.0.56:8643/healthz": dial tcp 10.134.0.56:8643: connect: connection refused
19:28:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-fail": the object has been modified; please apply your changes to the latest version and try again
19:28:50	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v
19:28:50	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4 from 0 to 1
19:28:51	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v	AddedInterface	Add eth0 [10.134.0.57/23] from ovn-kubernetes
19:28:55 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:28:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	InferenceServiceReady	InferenceService [isvc-paddle-runtime] is Ready
19:28:57 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v	Started	Started container storage-initializer
19:28:57 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v	Created	Created container: storage-initializer
19:28:57 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:28:57 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:29:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v	Killing	Stopping container storage-initializer
19:29:01 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-pass": the object has been modified; please apply your changes to the latest version and try again
19:29:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-pass": the object has been modified; please apply your changes to the latest version and try again
19:29:01	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c from 0 to 1
19:29:01	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42
19:29:01 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:29:01 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:29:02	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42	AddedInterface	Add eth0 [10.134.0.58/23] from ovn-kubernetes
19:29:02	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-ffwrv	Killing	Stopping container kserve-container
19:29:02	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-ffwrv	Killing	Stopping container kube-rbac-proxy
19:29:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42	Started	Started container storage-initializer
19:29:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42	Created	Created container: storage-initializer
19:29:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:29:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:29:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42	Started	Started container kserve-container
19:29:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42	Created	Created container: kserve-container
19:29:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42	Created	Created container: kube-rbac-proxy
19:29:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" already present on machine
19:29:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42	Started	Started container kube-rbac-proxy
19:29:05 (x7)	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-ffwrv	Unhealthy	Readiness probe failed: dial tcp 10.132.0.62:8080: connect: connection refused
19:29:05	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-ffwrv	Unhealthy	Readiness probe failed: Get "https://10.132.0.62:8643/healthz": dial tcp 10.132.0.62:8643: connect: connection refused
19:29:07	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-v2-kserve-predictor-64d4c74b4d	SuccessfulCreate	Created pod: isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv
19:29:07	kserve-ci-e2e-test	deployment-controller	isvc-paddle-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-v2-kserve-predictor-64d4c74b4d from 0 to 1
19:29:07	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
19:29:07	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
19:29:08	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv	Started	Started container storage-initializer
19:29:08	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:29:08	kserve-ci-e2e-test	multus	isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv	AddedInterface	Add eth0 [10.132.0.63/23] from ovn-kubernetes
19:29:08	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv	Created	Created container: storage-initializer
19:29:13	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv	Created	Created container: kube-rbac-proxy
19:29:13	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv	Started	Started container kube-rbac-proxy
19:29:13	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:29:13	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv	Started	Started container kserve-container
19:29:13	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv	Created	Created container: kserve-container
19:29:13	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv	Pulled	Container image "kserve/paddleserver:latest" already present on machine
19:29:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:29:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:29:52 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:29:52 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:30:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	InferenceServiceReady	InferenceService [isvc-paddle-v2-kserve] is Ready
19:30:01 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:30:07 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:30:07 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:30:08	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv	Killing	Stopping container kserve-container
19:30:08	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv	Killing	Stopping container kube-rbac-proxy
19:30:11 (x7)	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv	Unhealthy	Readiness probe failed: dial tcp 10.132.0.63:8080: connect: connection refused
19:30:11	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-pcjjv	Unhealthy	Readiness probe failed: Get "https://10.132.0.63:8643/healthz": dial tcp 10.132.0.63:8643: connect: connection refused
19:30:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	UpdateFailed	Failed to update status for InferenceService "isvc-pmml": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml": the object has been modified; please apply your changes to the latest version and try again
19:30:14	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6rsz5	Started	Started container storage-initializer
19:30:14	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6rsz5	Created	Created container: storage-initializer
19:30:14	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6rsz5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:30:14	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-predictor-8674795f59	SuccessfulCreate	Created pod: isvc-pmml-predictor-8674795f59-6rsz5
19:30:14	kserve-ci-e2e-test	deployment-controller	isvc-pmml-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-predictor-8674795f59 from 0 to 1
19:30:14	kserve-ci-e2e-test	multus	isvc-pmml-predictor-8674795f59-6rsz5	AddedInterface	Add eth0 [10.132.0.64/23] from ovn-kubernetes
19:30:16 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:30:16 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:30:19	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6rsz5	Pulling	Pulling image "kserve/pmmlserver:latest"
19:30:22 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:30:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-serving-pass] is Ready
19:30:26	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6rsz5	Created	Created container: kube-rbac-proxy
19:30:26	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6rsz5	Created	Created container: kserve-container
19:30:26	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6rsz5	Started	Started container kserve-container
19:30:26	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6rsz5	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:30:26	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6rsz5	Started	Started container kube-rbac-proxy
19:30:26	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6rsz5	Pulled	Successfully pulled image "kserve/pmmlserver:latest" in 7.177s (7.177s including waiting). Image size: 826884477 bytes.
19:30:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42	Killing	Stopping container kube-rbac-proxy
19:30:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42	Killing	Stopping container kserve-container
19:30:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42	Unhealthy	Readiness probe failed: Get "https://10.134.0.58:8643/healthz": dial tcp 10.134.0.58:8643: connect: connection refused
19:30:32 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42	Unhealthy	Readiness probe failed: dial tcp 10.134.0.58:8080: connect: connection refused
19:30:32	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc from 0 to 1
19:30:32	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch
19:30:33	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-fail": the object has been modified; please apply your changes to the latest version and try again
19:30:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
19:30:33 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-fail": the object has been modified; please apply your changes to the latest version and try again
19:30:34	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch	AddedInterface	Add eth0 [10.132.0.65/23] from ovn-kubernetes
19:30:39 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch	Started	Started container storage-initializer
19:30:39 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:30:39 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:30:39 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch	Created	Created container: storage-initializer
19:30:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-6f8b85cdfc-9kvch	Killing	Stopping container storage-initializer
19:30:43	kserve-ci-e2e-test	deployment-controller	isvc-pmml-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-runtime-predictor-885fc8777 from 0 to 1
19:30:44	kserve-ci-e2e-test	multus	isvc-pmml-runtime-predictor-885fc8777-fn2gk	AddedInterface	Add eth0 [10.132.0.66/23] from ovn-kubernetes
19:30:44	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-runtime-predictor-885fc8777	SuccessfulCreate	Created pod: isvc-pmml-runtime-predictor-885fc8777-fn2gk
19:30:44	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-fn2gk	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:30:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-runtime": the object has been modified; please apply your changes to the latest version and try again
19:30:44	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-fn2gk	Started	Started container storage-initializer
19:30:44 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-runtime": the object has been modified; please apply your changes to the latest version and try again
19:30:44	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-fn2gk	Created	Created container: storage-initializer
19:30:49	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-fn2gk	Started	Started container kserve-container
19:30:49	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-fn2gk	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:30:49	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-fn2gk	Created	Created container: kube-rbac-proxy
19:30:49	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-fn2gk	Started	Started container kube-rbac-proxy
19:30:49	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-fn2gk	Created	Created container: kserve-container
19:30:49	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-fn2gk	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
19:30:59 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:30:59 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:31:14 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:31:14 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:31:42 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6rsz5	Unhealthy	Readiness probe failed: dial tcp 10.132.0.64:8080: connect: connection refused
19:31:44 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:31:44 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:31:52 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:31:52	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	InferenceServiceReady	InferenceService [isvc-pmml] is Ready
19:31:55	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6rsz5	Killing	Stopping container kserve-container
19:31:55	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6rsz5	Killing	Stopping container kube-rbac-proxy
19:31:59	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-6rsz5	Unhealthy	Readiness probe failed: Get "https://10.132.0.64:8643/healthz": dial tcp 10.132.0.64:8643: connect: connection refused
19:32:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-a291d0	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-a291d0": the object has been modified; please apply your changes to the latest version and try again
19:32:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-a291d0	UpdateFailed	Failed to update status for InferenceService "isvc-primary-a291d0": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-a291d0": the object has been modified; please apply your changes to the latest version and try again
19:32:00	kserve-ci-e2e-test	replicaset-controller	isvc-primary-a291d0-predictor-7f984c747b	SuccessfulCreate	Created pod: isvc-primary-a291d0-predictor-7f984c747b-z922b
19:32:00	kserve-ci-e2e-test	deployment-controller	isvc-primary-a291d0-predictor	ScalingReplicaSet	Scaled up replica set isvc-primary-a291d0-predictor-7f984c747b from 0 to 1
19:32:01	kserve-ci-e2e-test	kubelet	isvc-primary-a291d0-predictor-7f984c747b-z922b	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:32:01	kserve-ci-e2e-test	kubelet	isvc-primary-a291d0-predictor-7f984c747b-z922b	Started	Started container storage-initializer
19:32:01	kserve-ci-e2e-test	multus	isvc-primary-a291d0-predictor-7f984c747b-z922b	AddedInterface	Add eth0 [10.134.0.59/23] from ovn-kubernetes
19:32:01	kserve-ci-e2e-test	kubelet	isvc-primary-a291d0-predictor-7f984c747b-z922b	Created	Created container: storage-initializer
19:32:06	kserve-ci-e2e-test	kubelet	isvc-primary-a291d0-predictor-7f984c747b-z922b	Started	Started container kube-rbac-proxy
19:32:06	kserve-ci-e2e-test	kubelet	isvc-primary-a291d0-predictor-7f984c747b-z922b	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1985" already present on machine
19:32:06	kserve-ci-e2e-test	kubelet	isvc-primary-a291d0-predictor-7f984c747b-z922b	Created	Created container: kube-rbac-proxy
19:32:06	kserve-ci-e2e-test	kubelet	isvc-primary-a291d0-predictor-7f984c747b-z922b	Started	Started container kserve-container
19:32:06	kserve-ci-e2e-test	kubelet	isvc-primary-a291d0-predictor-7f984c747b-z922b	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:32:06	kserve-ci-e2e-test	kubelet	isvc-primary-a291d0-predictor-7f984c747b-z922b	Created	Created container: kserve-container
19:32:14 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:32:14 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:32:18 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:32:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	InferenceServiceReady	InferenceService [isvc-pmml-runtime] is Ready
19:32:25	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-fn2gk	Killing	Stopping container kserve-container
19:32:25	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-fn2gk	Killing	Stopping container kube-rbac-proxy
19:32:27	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-fn2gk	Unhealthy	Readiness probe failed: Get "https://10.132.0.66:8643/healthz": dial tcp 10.132.0.66:8643: connect: connection refused
19:32:28 (x12)	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-fn2gk	Unhealthy	Readiness probe failed: dial tcp 10.132.0.66:8080: connect: connection refused
19:32:30 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-a291d0-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:32:30	kserve-ci-e2e-test	deployment-controller	isvc-pmml-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-v2-kserve-predictor-84cb55486c from 0 to 1
19:32:30	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-v2-kserve-predictor-84cb55486c	SuccessfulCreate	Created pod: isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8
19:32:30 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-a291d0-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:32:30	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
19:32:30	kserve-ci-e2e-test	multus	isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8	AddedInterface	Add eth0 [10.132.0.67/23] from ovn-kubernetes
19:32:31	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8	Created	Created container: storage-initializer
19:32:31	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8	Started	Started container storage-initializer
19:32:31	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:32:35	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8	Started	Started container kube-rbac-proxy
19:32:35	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
19:32:35	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8	Created	Created container: kserve-container
19:32:35	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8	Started	Started container kserve-container
19:32:35	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8	Created	Created container: kube-rbac-proxy
19:32:35	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:33:13 (x9)	kserve-ci-e2e-test	kubelet	isvc-primary-a291d0-predictor-7f984c747b-z922b	Unhealthy	Readiness probe failed: dial tcp 10.134.0.59:8080: connect: connection refused
19:33:15 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:33:15 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:33:17	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-a291d0	InferenceServiceReady	InferenceService [isvc-primary-a291d0] is Ready
19:33:17 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-a291d0	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:33:30 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-a291d0-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:33:30 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-a291d0-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:33:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-a291d0	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-a291d0": the object has been modified; please apply your changes to the latest version and try again
19:33:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-a291d0	UpdateFailed	Failed to update status for InferenceService "isvc-secondary-a291d0": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-a291d0": the object has been modified; please apply your changes to the latest version and try again
19:33:35	kserve-ci-e2e-test	deployment-controller	isvc-secondary-a291d0-predictor	ScalingReplicaSet	Scaled up replica set isvc-secondary-a291d0-predictor-64fbbf99f5 from 0 to 1
19:33:35	kserve-ci-e2e-test	replicaset-controller	isvc-secondary-a291d0-predictor-64fbbf99f5	SuccessfulCreate	Created pod: isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp
19:33:36	kserve-ci-e2e-test	kubelet	isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-secondary-a291d0-predictor-serving-cert" not found
19:33:36	kserve-ci-e2e-test	multus	isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp	AddedInterface	Add eth0 [10.134.0.60/23] from ovn-kubernetes
19:33:40 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp	Started	Started container storage-initializer
19:33:40 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a67c7de2a2df20ba196233af8aeab8be14c9a48c25488c0e22514b22fe3caba5" already present on machine
19:33:40 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp	Created	Created container: storage-initializer
19:33:44 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-a291d0	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:33:44	kserve-ci-e2e-test	kubelet	isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp	BackOff	Back-off restarting failed container storage-initializer in pod isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp_kserve-ci-e2e-test(e7c4cbf9-03b0-4a31-87e3-e0eee6661f37)
19:33:50	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-a291d0-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:33:50	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-a291d0-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:33:51	kserve-ci-e2e-test	kubelet	isvc-primary-a291d0-predictor-7f984c747b-z922b	Killing	Stopping container kserve-container
19:33:51	kserve-ci-e2e-test	kubelet	isvc-primary-a291d0-predictor-7f984c747b-z922b	Killing	Stopping container kube-rbac-proxy
19:33:53	kserve-ci-e2e-test	kubelet	isvc-primary-a291d0-predictor-7f984c747b-z922b	Unhealthy	Readiness probe failed: Get "https://10.134.0.59:8643/healthz": dial tcp 10.134.0.59:8643: connect: connection refused
19:34:00 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:34:00 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:34:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	InferenceServiceReady	InferenceService [isvc-pmml-v2-kserve] is Ready
19:34:04 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:34:11	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8	Killing	Stopping container kube-rbac-proxy
19:34:11	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8	Killing	Stopping container kserve-container
19:34:12	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8	Unhealthy	Readiness probe failed: Get "https://10.132.0.67:8643/healthz": dial tcp 10.132.0.67:8643: connect: connection refused
19:34:14 (x12)	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-68gk8	Unhealthy	Readiness probe failed: dial tcp 10.132.0.67:8080: connect: connection refused