Events

Time	Namespace	Component	RelatedObject	Reason	Message
19:19:34	kserve-ci-e2e-test		isvc-canary-rollback-predictor-665c868b97-tm5sv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-tm5sv to ip-10-0-130-142.ec2.internal
19:33:25	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj to ip-10-0-130-142.ec2.internal
19:29:52	kserve-ci-e2e-test		isvc-paddle-predictor-6d5c9fd4f6-q7648	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q7648 to ip-10-0-130-142.ec2.internal
19:15:53	kserve-ci-e2e-test		isvc-tensorflow-predictor-547dfc5669-ssm8l	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l to ip-10-0-128-180.ec2.internal
19:11:37	kserve-ci-e2e-test		isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb to ip-10-0-128-180.ec2.internal
19:27:31	kserve-ci-e2e-test		isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s to ip-10-0-128-180.ec2.internal
18:58:44	kserve-ci-e2e-test		isvc-logger-predictor-6cb66b5bc5-h6kb7	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7 to ip-10-0-128-180.ec2.internal
19:16:40	kserve-ci-e2e-test		isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2 to ip-10-0-130-142.ec2.internal
18:55:09	kserve-ci-e2e-test		isvc-predictive-sklearn-predictor-5f849d69df-c6hpz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz to ip-10-0-128-180.ec2.internal
19:26:11	kserve-ci-e2e-test		isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh to ip-10-0-128-180.ec2.internal
19:32:14	kserve-ci-e2e-test		isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv to ip-10-0-130-142.ec2.internal
19:17:26	kserve-ci-e2e-test		isvc-triton-predictor-5d7db58676-gpgpq	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq to ip-10-0-128-180.ec2.internal
19:27:06	kserve-ci-e2e-test		isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj to ip-10-0-130-142.ec2.internal
19:00:19	kserve-ci-e2e-test		isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d to ip-10-0-130-142.ec2.internal
19:19:53	kserve-ci-e2e-test		isvc-xgboost-predictor-7566d65bf7-ccl6q	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ccl6q to ip-10-0-130-142.ec2.internal
19:24:35	kserve-ci-e2e-test		isvc-lightgbm-runtime-predictor-577f7db7cb-888mm	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm to ip-10-0-128-180.ec2.internal
19:23:19	kserve-ci-e2e-test		isvc-xgboost-runtime-predictor-54dd488c85-l49vq	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-l49vq to ip-10-0-130-142.ec2.internal
19:21:29	kserve-ci-e2e-test		isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj to ip-10-0-128-180.ec2.internal
19:25:50	kserve-ci-e2e-test		isvc-xgboost-v2-predictor-fcc668bc-8t6kd	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-8t6kd to ip-10-0-130-142.ec2.internal
19:24:44	kserve-ci-e2e-test		isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b to ip-10-0-128-180.ec2.internal
18:58:24	kserve-ci-e2e-test		message-dumper-predictor-6575bcfdfd-nb6l2	Scheduled	Successfully assigned kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2 to ip-10-0-128-180.ec2.internal
19:23:00	kserve-ci-e2e-test		isvc-lightgbm-predictor-54f85ddf88-fvgb2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-fvgb2 to ip-10-0-130-142.ec2.internal
19:07:12	kserve-ci-e2e-test		isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx to ip-10-0-130-142.ec2.internal
19:11:25	kserve-ci-e2e-test		isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p to ip-10-0-130-142.ec2.internal
19:12:53	kserve-ci-e2e-test		isvc-canary-create-predictor-6cc94c6f4f-lcngg	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-lcngg to ip-10-0-130-142.ec2.internal
19:12:53	kserve-ci-e2e-test		isvc-canary-create-predictor-6cc94c6f4f-qkb8c	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c to ip-10-0-128-180.ec2.internal
18:58:43	kserve-ci-e2e-test		isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt to ip-10-0-128-180.ec2.internal
18:54:54	kserve-ci-e2e-test		isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc to ip-10-0-128-180.ec2.internal
18:57:08	kserve-ci-e2e-test		isvc-predictive-xgboost-predictor-695f978978-7xp24	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24 to ip-10-0-128-180.ec2.internal
19:07:44	kserve-ci-e2e-test		sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz	Scheduled	Successfully assigned kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz to ip-10-0-128-180.ec2.internal
19:21:19	kserve-ci-e2e-test		isvc-canary-stop-v2-predictor-b7c9ff648-65lhc	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b7c9ff648-65lhc to ip-10-0-130-142.ec2.internal
19:12:13	kserve-ci-e2e-test		isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg to ip-10-0-130-142.ec2.internal
19:21:19	kserve-ci-e2e-test		isvc-canary-stop-predictor-7bb9c548b5-xr4pd	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd to ip-10-0-128-180.ec2.internal
19:22:24	kserve-ci-e2e-test		xgboost-v2-mlserver-predictor-59944bcf4c-d5prm	Scheduled	Successfully assigned kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm to ip-10-0-128-180.ec2.internal
19:21:20	kserve-ci-e2e-test		isvc-canary-stop-predictor-7bb9c548b5-48h5k	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-48h5k to ip-10-0-130-142.ec2.internal
19:02:24	kserve-ci-e2e-test		isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8 to ip-10-0-130-142.ec2.internal
19:17:44	kserve-ci-e2e-test		isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt to ip-10-0-130-142.ec2.internal
19:35:07	kserve-ci-e2e-test		isvc-primary-404a29-predictor-8574c4b8bf-kvtlt	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-primary-404a29-predictor-8574c4b8bf-kvtlt to ip-10-0-130-142.ec2.internal
19:28:27	kserve-ci-e2e-test		isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc to ip-10-0-128-180.ec2.internal
19:10:49	kserve-ci-e2e-test		isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv to ip-10-0-128-180.ec2.internal
19:06:08	kserve-ci-e2e-test		isvc-sklearn-predictor-d96586f44-qrtkr	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr to ip-10-0-128-180.ec2.internal
19:03:01	kserve-ci-e2e-test		isvc-autogluon-v2-predictor-9684b4cd9-hvzvj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-hvzvj to ip-10-0-130-142.ec2.internal
19:36:43	kserve-ci-e2e-test		isvc-secondary-404a29-predictor-544c4b6766-nlpb8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-secondary-404a29-predictor-544c4b6766-nlpb8 to ip-10-0-130-142.ec2.internal
19:12:53	kserve-ci-e2e-test		isvc-canary-create-v2-predictor-7855d67596-sgvxp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp to ip-10-0-128-180.ec2.internal
19:17:44	kserve-ci-e2e-test		isvc-canary-rollback-predictor-665c868b97-vqw4s	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s to ip-10-0-128-180.ec2.internal
19:09:34	kserve-ci-e2e-test		isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z to ip-10-0-130-142.ec2.internal
19:03:59	kserve-ci-e2e-test		isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t to ip-10-0-128-180.ec2.internal
19:00:19	kserve-ci-e2e-test		isvc-autogluon-v1-predictor-574b5b7495-png7f	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f to ip-10-0-128-180.ec2.internal
19:14:38	kserve-ci-e2e-test		isvc-canary-promote-predictor-b9b98bcd9-bj7dj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj to ip-10-0-128-180.ec2.internal
18:54:54	kserve-ci-e2e-test		isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f to ip-10-0-128-180.ec2.internal
19:17:44	kserve-ci-e2e-test		isvc-canary-rollback-predictor-665c868b97-8fg87	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-8fg87 to ip-10-0-130-142.ec2.internal
18:56:49	kserve-ci-e2e-test		isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs to ip-10-0-128-180.ec2.internal
19:12:42	kserve-ci-e2e-test		isvc-sklearn-v2-predictor-7b474946d-56mbs	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs to ip-10-0-128-180.ec2.internal
19:33:21	kserve-ci-e2e-test		isvc-pmml-predictor-8674795f59-hcrpq	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-hcrpq to ip-10-0-130-142.ec2.internal
19:14:18	kserve-ci-e2e-test		isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh to ip-10-0-130-142.ec2.internal
19:14:38	kserve-ci-e2e-test		isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m to ip-10-0-130-142.ec2.internal
19:16:09	kserve-ci-e2e-test		isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x to ip-10-0-128-180.ec2.internal
19:31:54	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w to ip-10-0-130-142.ec2.internal
19:31:08	kserve-ci-e2e-test		isvc-paddle-runtime-predictor-58b99fff76-g5mt6	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-g5mt6 to ip-10-0-130-142.ec2.internal
19:28:39	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44 to ip-10-0-130-142.ec2.internal
19:33:36	kserve-ci-e2e-test		isvc-pmml-runtime-predictor-885fc8777-qz2qm	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-qz2qm to ip-10-0-130-142.ec2.internal
19:30:10	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7 to ip-10-0-128-180.ec2.internal
19:30:21	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt to ip-10-0-130-142.ec2.internal
19:31:43	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5 to ip-10-0-128-180.ec2.internal
19:27:15	kserve-ci-e2e-test		isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx to ip-10-0-130-142.ec2.internal
19:14:39	kserve-ci-e2e-test		isvc-canary-promote-predictor-b9b98bcd9-tdnsw	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-tdnsw to ip-10-0-130-142.ec2.internal
19:35:22	kserve-ci-e2e-test		isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx to ip-10-0-130-142.ec2.internal
18:54:53	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-predictor-7946b7fbfb from 0 to 1
18:54:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
18:54:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
18:54:54	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-predictor-7946b7fbfb	SuccessfulCreate	Created pod: isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f
18:54:54	kserve-ci-e2e-test	kubelet	isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-init-fail-7e78f2-predictor-serving-cert" not found
18:54:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
18:54:54	kserve-ci-e2e-test	replicaset-controller	isvc-init-fail-7e78f2-predictor-67df985b46	SuccessfulCreate	Created pod: isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc
18:54:54	kserve-ci-e2e-test	deployment-controller	isvc-init-fail-7e78f2-predictor	ScalingReplicaSet	Scaled up replica set isvc-init-fail-7e78f2-predictor-67df985b46 from 0 to 1
18:54:55	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f	AddedInterface	Add eth0 [10.132.0.27/23] from ovn-kubernetes
18:54:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
18:54:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f	Started	Started container storage-initializer
18:54:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f	Created	Created container: storage-initializer
18:54:55	kserve-ci-e2e-test	multus	isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc	AddedInterface	Add eth0 [10.132.0.28/23] from ovn-kubernetes
18:55:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986"
18:55:00 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
18:55:00 (x8)	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-7e78f2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:55:00 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc	Started	Started container storage-initializer
18:55:00 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc	Created	Created container: storage-initializer
18:55:04	kserve-ci-e2e-test	kubelet	isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc	Killing	Stopping container storage-initializer
18:55:09	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-predictor-5f849d69df from 0 to 1
18:55:09	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-predictor-5f849d69df	SuccessfulCreate	Created pod: isvc-predictive-sklearn-predictor-5f849d69df-c6hpz
18:55:09	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn": the object has been modified; please apply your changes to the latest version and try again
18:55:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" in 13.021s (13.021s including waiting). Image size: 1580859293 bytes.
18:55:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f	Created	Created container: kserve-container
18:55:13	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-c6hpz	Started	Started container storage-initializer
18:55:13	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-c6hpz	Created	Created container: storage-initializer
18:55:13	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-c6hpz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
18:55:13	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-predictor-5f849d69df-c6hpz	AddedInterface	Add eth0 [10.132.0.29/23] from ovn-kubernetes
18:55:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
18:55:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f	Started	Started container kserve-container
18:55:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f	Pulling	Pulling image "quay.io/opendatahub/kserve-agent@sha256:0dffbff0f1e03e1de327a5bd3cf75afe449e165a6aea5f2318608e71c8ad9441"
18:55:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f	Started	Started container kube-rbac-proxy
18:55:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f	Created	Created container: kube-rbac-proxy
18:55:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.299s (2.299s including waiting). Image size: 211946088 bytes.
18:55:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:0dffbff0f1e03e1de327a5bd3cf75afe449e165a6aea5f2318608e71c8ad9441" in 2.491s (2.491s including waiting). Image size: 239964773 bytes.
18:55:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f	Created	Created container: agent
18:55:18	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-c6hpz	Pulling	Pulling image "kserve/predictiveserver:latest"
18:55:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f	Started	Started container agent
18:55:24 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:55:24 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:55:40	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-c6hpz	Created	Created container: kube-rbac-proxy
18:55:40	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-c6hpz	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:55:40	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-c6hpz	Started	Started container kube-rbac-proxy
18:55:40	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-c6hpz	Started	Started container kserve-container
18:55:40	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-c6hpz	Created	Created container: kserve-container
18:55:40	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-c6hpz	Pulled	Successfully pulled image "kserve/predictiveserver:latest" in 22.049s (22.049s including waiting). Image size: 2434679947 bytes.
18:56:09 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:56:09 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:56:39 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:56:39 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:56:41 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:56:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InferenceServiceReady	InferenceService [isvc-sklearn-batcher] is Ready
18:56:48 (x9)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-c6hpz	Unhealthy	Readiness probe failed: dial tcp 10.132.0.29:8080: connect: connection refused
18:56:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs	Created	Created container: storage-initializer
18:56:49	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-custom-predictor-d9f644498	SuccessfulCreate	Created pod: isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs
18:56:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs	Started	Started container storage-initializer
18:56:49	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs	AddedInterface	Add eth0 [10.132.0.30/23] from ovn-kubernetes
18:56:49 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher-custom": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher-custom": the object has been modified; please apply your changes to the latest version and try again
18:56:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
18:56:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f	Killing	Stopping container kserve-container
18:56:49	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-custom-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-custom-predictor-d9f644498 from 0 to 1
18:56:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher-custom": the object has been modified; please apply your changes to the latest version and try again
18:56:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f	Killing	Stopping container kube-rbac-proxy
18:56:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f	Killing	Stopping container agent
18:56:51 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
18:56:51 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f	Unhealthy	Readiness probe failed: dial tcp 10.132.0.27:8080: connect: connection refused
18:56:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs	Created	Created container: agent
18:56:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:0dffbff0f1e03e1de327a5bd3cf75afe449e165a6aea5f2318608e71c8ad9441" already present on machine
18:56:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" already present on machine
18:56:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs	Started	Started container agent
18:56:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs	Started	Started container kube-rbac-proxy
18:56:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs	Created	Created container: kserve-container
18:56:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs	Started	Started container kserve-container
18:56:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:56:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs	Created	Created container: kube-rbac-proxy
18:56:54 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:56:54 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:56:55 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f	Unhealthy	Readiness probe failed: Get "https://10.132.0.27:8643/healthz": dial tcp 10.132.0.27:8643: connect: connection refused
18:56:58	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	InferenceServiceReady	InferenceService [isvc-predictive-sklearn] is Ready
18:56:58 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:57:03	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-c6hpz	Killing	Stopping container kube-rbac-proxy
18:57:03	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-c6hpz	Killing	Stopping container kserve-container
18:57:03	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-c6hpz	Unhealthy	Readiness probe failed: Get "https://10.132.0.29:8643/healthz": dial tcp 10.132.0.29:8643: connect: connection refused
18:57:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost": the object has been modified; please apply your changes to the latest version and try again
18:57:08	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-predictor-695f978978	SuccessfulCreate	Created pod: isvc-predictive-xgboost-predictor-695f978978-7xp24
18:57:08	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-predictor-695f978978 from 0 to 1
18:57:09	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-7xp24	Started	Started container storage-initializer
18:57:09	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-7xp24	Created	Created container: storage-initializer
18:57:09	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-7xp24	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
18:57:09	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-predictor-695f978978-7xp24	AddedInterface	Add eth0 [10.132.0.31/23] from ovn-kubernetes
18:57:13	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-7xp24	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:57:13	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-7xp24	Created	Created container: kube-rbac-proxy
18:57:13	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-7xp24	Started	Started container kserve-container
18:57:13	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-7xp24	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
18:57:13	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-7xp24	Created	Created container: kserve-container
18:57:13	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-7xp24	Started	Started container kube-rbac-proxy
18:57:34 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:57:34 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:57:53 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:57:53 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:58:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	InferenceServiceReady	InferenceService [isvc-sklearn-batcher-custom] is Ready
18:58:11 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:58:19 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:58:19 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:58:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:58:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:58:24	kserve-ci-e2e-test	multus	message-dumper-predictor-6575bcfdfd-nb6l2	AddedInterface	Add eth0 [10.132.0.32/23] from ovn-kubernetes
18:58:24	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	UpdateFailed	Failed to update status for InferenceService "message-dumper": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper": the object has been modified; please apply your changes to the latest version and try again
18:58:24	kserve-ci-e2e-test	replicaset-controller	message-dumper-predictor-6575bcfdfd	SuccessfulCreate	Created pod: message-dumper-predictor-6575bcfdfd-nb6l2
18:58:24	kserve-ci-e2e-test	deployment-controller	message-dumper-predictor	ScalingReplicaSet	Scaled up replica set message-dumper-predictor-6575bcfdfd from 0 to 1
18:58:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs	Killing	Stopping container kube-rbac-proxy
18:58:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs	Killing	Stopping container agent
18:58:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs	Killing	Stopping container kserve-container
18:58:24	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-nb6l2	Pulling	Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display"
18:58:26	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-nb6l2	Started	Started container kube-rbac-proxy
18:58:26	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-nb6l2	Created	Created container: kube-rbac-proxy
18:58:26	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-nb6l2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:58:26	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-nb6l2	Started	Started container kserve-container
18:58:26	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-nb6l2	Created	Created container: kserve-container
18:58:26	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-nb6l2	Pulled	Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.204s (1.204s including waiting). Image size: 14813193 bytes.
18:58:30	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	InferenceServiceReady	InferenceService [isvc-predictive-xgboost] is Ready
18:58:30 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:58:31 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs	Unhealthy	Readiness probe failed: dial tcp 10.132.0.30:5000: connect: connection refused
18:58:31 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
18:58:35 (x7)	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:58:35	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	InferenceServiceReady	InferenceService [message-dumper] is Ready
18:58:38	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-7xp24	Killing	Stopping container kube-rbac-proxy
18:58:38	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-7xp24	Killing	Stopping container kserve-container
18:58:40	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-7xp24	Unhealthy	Readiness probe failed: Get "https://10.132.0.31:8643/healthz": dial tcp 10.132.0.31:8643: connect: connection refused
18:58:40 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-7xp24	Unhealthy	Readiness probe failed: dial tcp 10.132.0.31:8080: connect: connection refused
18:58:41 (x4)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs	Unhealthy	Readiness probe failed: Get "https://10.132.0.30:8643/healthz": dial tcp 10.132.0.30:8643: connect: connection refused
18:58:43	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
18:58:43	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm": the object has been modified; please apply your changes to the latest version and try again
18:58:43	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-predictor-ff56bcbb5	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt
18:58:43	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-predictor-ff56bcbb5 from 0 to 1
18:58:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	UpdateFailed	Failed to update status for InferenceService "isvc-logger": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger": the object has been modified; please apply your changes to the latest version and try again
18:58:44	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt	Started	Started container storage-initializer
18:58:44	kserve-ci-e2e-test	deployment-controller	isvc-logger-predictor	ScalingReplicaSet	Scaled up replica set isvc-logger-predictor-6cb66b5bc5 from 0 to 1
18:58:44	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt	Created	Created container: storage-initializer
18:58:44	kserve-ci-e2e-test	replicaset-controller	isvc-logger-predictor-6cb66b5bc5	SuccessfulCreate	Created pod: isvc-logger-predictor-6cb66b5bc5-h6kb7
18:58:44	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
18:58:44	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6cb66b5bc5-h6kb7	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-logger-predictor-serving-cert" not found
18:58:44	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt	AddedInterface	Add eth0 [10.132.0.33/23] from ovn-kubernetes
18:58:45	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6cb66b5bc5-h6kb7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
18:58:45	kserve-ci-e2e-test	multus	isvc-logger-predictor-6cb66b5bc5-h6kb7	AddedInterface	Add eth0 [10.132.0.34/23] from ovn-kubernetes
18:58:45	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6cb66b5bc5-h6kb7	Created	Created container: storage-initializer
18:58:45	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6cb66b5bc5-h6kb7	Started	Started container storage-initializer
18:58:49	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
18:58:49	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt	Started	Started container kube-rbac-proxy
18:58:49	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt	Created	Created container: kserve-container
18:58:49	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt	Created	Created container: kube-rbac-proxy
18:58:49	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:58:49	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt	Started	Started container kserve-container
18:58:50	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6cb66b5bc5-h6kb7	Created	Created container: agent
18:58:50	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6cb66b5bc5-h6kb7	Started	Started container kube-rbac-proxy
18:58:50	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6cb66b5bc5-h6kb7	Started	Started container agent
18:58:50	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6cb66b5bc5-h6kb7	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" already present on machine
18:58:50	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6cb66b5bc5-h6kb7	Started	Started container kserve-container
18:58:50	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6cb66b5bc5-h6kb7	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:0dffbff0f1e03e1de327a5bd3cf75afe449e165a6aea5f2318608e71c8ad9441" already present on machine
18:58:50	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6cb66b5bc5-h6kb7	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:58:50	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6cb66b5bc5-h6kb7	Created	Created container: kserve-container
18:58:50	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6cb66b5bc5-h6kb7	Created	Created container: kube-rbac-proxy
18:58:54 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:58:54 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:58:58	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:58:58	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:58:59	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:58:59	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:59:58 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:59:58 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:00:07	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm] is Ready
19:00:07 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:00:12	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	InferenceServiceReady	InferenceService [isvc-logger] is Ready
19:00:12 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:00:13	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt	Killing	Stopping container kube-rbac-proxy
19:00:13	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt	Killing	Stopping container kserve-container
19:00:14 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:00:14 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:00:17	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt	Unhealthy	Readiness probe failed: Get "https://10.132.0.33:8643/healthz": dial tcp 10.132.0.33:8643: connect: connection refused
19:00:17 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt	Unhealthy	Readiness probe failed: dial tcp 10.132.0.33:8080: connect: connection refused
19:00:18	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-v2-predictor-745f9c8c9f from 0 to 1
19:00:19	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6cb66b5bc5-h6kb7	Killing	Stopping container agent
19:00:19	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-nb6l2	Killing	Stopping container kube-rbac-proxy
19:00:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v1": the object has been modified; please apply your changes to the latest version and try again
19:00:19	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6cb66b5bc5-h6kb7	Killing	Stopping container kserve-container
19:00:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
19:00:19 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
19:00:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v1": the object has been modified; please apply your changes to the latest version and try again
19:00:19	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-nb6l2	Killing	Stopping container kserve-container
19:00:19	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d	AddedInterface	Add eth0 [10.134.0.32/23] from ovn-kubernetes
19:00:19	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d"
19:00:19	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-v2-predictor-745f9c8c9f	SuccessfulCreate	Created pod: isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d
19:00:19	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v1-predictor-574b5b7495	SuccessfulCreate	Created pod: isvc-autogluon-v1-predictor-574b5b7495-png7f
19:00:19	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v1-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v1-predictor-574b5b7495 from 0 to 1
19:00:19	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6cb66b5bc5-h6kb7	Killing	Stopping container kube-rbac-proxy
19:00:20	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-png7f	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:00:20	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-png7f	Started	Started container storage-initializer
19:00:20	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-png7f	Created	Created container: storage-initializer
19:00:20	kserve-ci-e2e-test	multus	isvc-autogluon-v1-predictor-574b5b7495-png7f	AddedInterface	Add eth0 [10.132.0.35/23] from ovn-kubernetes
19:00:22 (x11)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6cb66b5bc5-h6kb7	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
19:00:22 (x12)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6cb66b5bc5-h6kb7	Unhealthy	Readiness probe failed: dial tcp 10.132.0.34:8080: connect: connection refused
19:00:23	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6cb66b5bc5-h6kb7	Unhealthy	Readiness probe failed: Get "https://10.132.0.34:8643/healthz": dial tcp 10.132.0.34:8643: connect: connection refused
19:00:23	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" in 4.291s (4.291s including waiting). Image size: 308172227 bytes.
19:00:24	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d	Created	Created container: storage-initializer
19:00:24	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d	Started	Started container storage-initializer
19:00:25	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-png7f	Pulling	Pulling image "kserve/autogluonserver:latest"
19:00:28	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d	Pulling	Pulling image "kserve/predictiveserver:latest"
19:00:50	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
19:00:50	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d	Started	Started container kserve-container
19:00:50	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d	Created	Created container: kserve-container
19:00:50	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d	Pulled	Successfully pulled image "kserve/predictiveserver:latest" in 21.583s (21.583s including waiting). Image size: 2434679947 bytes.
19:00:53	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 3.499s (3.499s including waiting). Image size: 211946088 bytes.
19:00:54	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d	Created	Created container: kube-rbac-proxy
19:00:54	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d	Started	Started container kube-rbac-proxy
19:01:34 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:01:34 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:02:04 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:02:04 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:02:11 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:02:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	InferenceServiceReady	InferenceService [isvc-predictive-sklearn-v2] is Ready
19:02:19	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d	Killing	Stopping container kserve-container
19:02:19	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d	Killing	Stopping container kube-rbac-proxy
19:02:20	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d	Unhealthy	Readiness probe failed: Get "https://10.134.0.32:8643/healthz": dial tcp 10.134.0.32:8643: connect: connection refused
19:02:20 (x6)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-wrz6d	Unhealthy	Readiness probe failed: Get "http://10.134.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready": dial tcp 10.134.0.32:8080: connect: connection refused
19:02:22	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-png7f	Created	Created container: kserve-container
19:02:22	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-png7f	Started	Started container kserve-container
19:02:22	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-png7f	Started	Started container kube-rbac-proxy
19:02:22	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-png7f	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:02:22	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-png7f	Pulled	Successfully pulled image "kserve/autogluonserver:latest" in 1m56.631s (1m56.631s including waiting). Image size: 8945772325 bytes.
19:02:22	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-png7f	Created	Created container: kube-rbac-proxy
19:02:24	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
19:02:24	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-v2-predictor-868ff5dd6d	SuccessfulCreate	Created pod: isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8
19:02:24	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
19:02:24	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-v2-predictor-868ff5dd6d from 0 to 1
19:02:25	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8	AddedInterface	Add eth0 [10.134.0.33/23] from ovn-kubernetes
19:02:25	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:02:25	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8	Started	Started container storage-initializer
19:02:25	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8	Created	Created container: storage-initializer
19:02:30	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8	Started	Started container kserve-container
19:02:30 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-png7f	Unhealthy	Readiness probe failed: dial tcp 10.132.0.35:8080: connect: connection refused
19:02:30	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8	Started	Started container kube-rbac-proxy
19:02:30	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
19:02:30	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8	Created	Created container: kserve-container
19:02:30	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8	Created	Created container: kube-rbac-proxy
19:02:30	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:02:40 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:02:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	InferenceServiceReady	InferenceService [isvc-autogluon-v1] is Ready
19:02:49 (x10)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:02:49 (x10)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:02:54 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:02:54 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:02:55	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-png7f	Killing	Stopping container kube-rbac-proxy
19:02:55	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-png7f	Killing	Stopping container kserve-container
19:03:01	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-predictor-9684b4cd9	SuccessfulCreate	Created pod: isvc-autogluon-v2-predictor-9684b4cd9-hvzvj
19:03:01	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-predictor-9684b4cd9 from 0 to 1
19:03:01	kserve-ci-e2e-test	multus	isvc-autogluon-v2-predictor-9684b4cd9-hvzvj	AddedInterface	Add eth0 [10.134.0.34/23] from ovn-kubernetes
19:03:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2": the object has been modified; please apply your changes to the latest version and try again
19:03:01	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-hvzvj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:03:01	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-hvzvj	Created	Created container: storage-initializer
19:03:01	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-hvzvj	Started	Started container storage-initializer
19:03:15	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-hvzvj	Pulling	Pulling image "kserve/autogluonserver:latest"
19:03:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:03:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:03:48	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	InferenceServiceReady	InferenceService [isvc-predictive-xgboost-v2] is Ready
19:03:48 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:03:54	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8	Killing	Stopping container kube-rbac-proxy
19:03:54	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8	Killing	Stopping container kserve-container
19:03:58 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8	Unhealthy	Readiness probe failed: Get "http://10.134.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready": dial tcp 10.134.0.33:8080: connect: connection refused
19:03:58	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nh4f8	Unhealthy	Readiness probe failed: Get "https://10.134.0.33:8643/healthz": dial tcp 10.134.0.33:8643: connect: connection refused
19:03:59	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-v2-predictor-545b7cc587	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t
19:03:59	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm-v2": the object has been modified; please apply your changes to the latest version and try again
19:03:59	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-v2-predictor-545b7cc587 from 0 to 1
19:04:00	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:04:00	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t	AddedInterface	Add eth0 [10.132.0.36/23] from ovn-kubernetes
19:04:00	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t	Created	Created container: storage-initializer
19:04:00	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t	Started	Started container storage-initializer
19:04:04	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t	Started	Started container kserve-container
19:04:04	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
19:04:04	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t	Created	Created container: kserve-container
19:04:04	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t	Started	Started container kube-rbac-proxy
19:04:04	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t	Created	Created container: kube-rbac-proxy
19:04:04	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:04:29 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:04:29 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:05:18	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-hvzvj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:05:18	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-hvzvj	Started	Started container kube-rbac-proxy
19:05:18	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-hvzvj	Created	Created container: kube-rbac-proxy
19:05:18	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-hvzvj	Pulled	Successfully pulled image "kserve/autogluonserver:latest" in 2m2.723s (2m2.723s including waiting). Image size: 8945772325 bytes.
19:05:18	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-hvzvj	Created	Created container: kserve-container
19:05:18	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-hvzvj	Started	Started container kserve-container
19:05:21	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm-v2] is Ready
19:05:22 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:05:29 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:05:29	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t	Killing	Stopping container kserve-container
19:05:29 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:05:29	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t	Killing	Stopping container kube-rbac-proxy
19:05:31	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t	Unhealthy	Readiness probe failed: Get "https://10.132.0.36:8643/healthz": dial tcp 10.132.0.36:8643: connect: connection refused
19:05:31 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t	Unhealthy	Readiness probe failed: Get "http://10.132.0.36:8080/v2/models/isvc-predictive-lightgbm-v2/ready": dial tcp 10.132.0.36:8080: connect: connection refused
19:05:34	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-scheduler-predictor-567777fff4	SuccessfulCreate	Created pod: isvc-sklearn-scheduler-predictor-567777fff4-klxkh
19:05:34	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-scheduler-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-scheduler-predictor-567777fff4 from 0 to 1
19:05:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-scheduler": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
19:05:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
19:05:35 (x6)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:05:46 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:05:46 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:06:01	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:06:08	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-predictor-d96586f44 from 0 to 1
19:06:08 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn": the object has been modified; please apply your changes to the latest version and try again
19:06:08	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-predictor-d96586f44	SuccessfulCreate	Created pod: isvc-sklearn-predictor-d96586f44-qrtkr
19:06:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn": the object has been modified; please apply your changes to the latest version and try again
19:06:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d96586f44-qrtkr	Started	Started container storage-initializer
19:06:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d96586f44-qrtkr	Created	Created container: storage-initializer
19:06:09	kserve-ci-e2e-test	multus	isvc-sklearn-predictor-d96586f44-qrtkr	AddedInterface	Add eth0 [10.132.0.39/23] from ovn-kubernetes
19:06:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d96586f44-qrtkr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:06:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d96586f44-qrtkr	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" already present on machine
19:06:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d96586f44-qrtkr	Started	Started container kube-rbac-proxy
19:06:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d96586f44-qrtkr	Started	Started container kserve-container
19:06:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d96586f44-qrtkr	Created	Created container: kserve-container
19:06:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d96586f44-qrtkr	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:06:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d96586f44-qrtkr	Created	Created container: kube-rbac-proxy
19:06:16 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:06:38 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:06:38 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:06:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	InferenceServiceReady	InferenceService [isvc-autogluon-v2] is Ready
19:06:55 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:07:06	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-hvzvj	Unhealthy	Readiness probe failed: Get "http://10.134.0.34:8080/v2/models/isvc-autogluon-v2/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
19:07:07	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-hvzvj	Killing	Stopping container kube-rbac-proxy
19:07:07	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-hvzvj	Killing	Stopping container kserve-container
19:07:11	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-hvzvj	Unhealthy	Readiness probe failed: Get "https://10.134.0.34:8643/healthz": dial tcp 10.134.0.34:8643: connect: connection refused
19:07:12	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2-variants": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-variants": the object has been modified; please apply your changes to the latest version and try again
19:07:12	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
19:07:12	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-variants-predictor-5c889d8844	SuccessfulCreate	Created pod: isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx
19:07:12	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-variants-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-variants-predictor-5c889d8844 from 0 to 1
19:07:13	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:07:13	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx	Started	Started container storage-initializer
19:07:13	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx	Created	Created container: storage-initializer
19:07:13	kserve-ci-e2e-test	multus	isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx	AddedInterface	Add eth0 [10.134.0.35/23] from ovn-kubernetes
19:07:32	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	InferenceServiceReady	InferenceService [isvc-sklearn] is Ready
19:07:32 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:07:38 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:07:38 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:07:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d96586f44-qrtkr	Killing	Stopping container kube-rbac-proxy
19:07:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d96586f44-qrtkr	Killing	Stopping container kserve-container
19:07:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d96586f44-qrtkr	Unhealthy	Readiness probe failed: Get "https://10.132.0.39:8643/healthz": dial tcp 10.132.0.39:8643: connect: connection refused
19:07:42 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d96586f44-qrtkr	Unhealthy	Readiness probe failed: dial tcp 10.132.0.39:8080: connect: connection refused
19:07:43	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx	Started	Started container kserve-container
19:07:43	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
19:07:43	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx	Created	Created container: kserve-container
19:07:43	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx	Created	Created container: kube-rbac-proxy
19:07:43	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:07:43	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx	Started	Started container kube-rbac-proxy
19:07:44	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "sklearn-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "sklearn-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
19:07:44	kserve-ci-e2e-test	deployment-controller	sklearn-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set sklearn-v2-mlserver-predictor-7f568cdc6b from 0 to 1
19:07:44	kserve-ci-e2e-test	replicaset-controller	sklearn-v2-mlserver-predictor-7f568cdc6b	SuccessfulCreate	Created pod: sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz
19:07:44	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
19:07:45	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz	Created	Created container: storage-initializer
19:07:45	kserve-ci-e2e-test	multus	sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz	AddedInterface	Add eth0 [10.132.0.40/23] from ovn-kubernetes
19:07:45	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz	Started	Started container storage-initializer
19:07:45	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:07:49	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz	Pulling	Pulling image "docker.io/seldonio/mlserver:1.7.1"
19:08:12 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:08:12 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:09:13	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	InferenceServiceReady	InferenceService [isvc-autogluon-v2-variants] is Ready
19:09:13 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:09:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx	Unhealthy	Readiness probe failed: Get "http://10.134.0.35:8080/v2/models/isvc-autogluon-v2-variants/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
19:09:27 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:09:27 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:09:29	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx	Killing	Stopping container kserve-container
19:09:29	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-wnkpx	Killing	Stopping container kube-rbac-proxy
19:09:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-noslash": the object has been modified; please apply your changes to the latest version and try again
19:09:34	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-noslash-predictor-5499c79f9f	SuccessfulCreate	Created pod: isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z
19:09:34 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2-noslash": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-noslash": the object has been modified; please apply your changes to the latest version and try again
19:09:34	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-noslash-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-noslash-predictor-5499c79f9f from 0 to 1
19:09:35	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z	Created	Created container: storage-initializer
19:09:35	kserve-ci-e2e-test	multus	isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z	AddedInterface	Add eth0 [10.134.0.36/23] from ovn-kubernetes
19:09:35	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z	Started	Started container storage-initializer
19:09:35	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:09:40	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z	Created	Created container: kserve-container
19:09:40	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z	Started	Started container kube-rbac-proxy
19:09:40	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
19:09:40	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z	Created	Created container: kube-rbac-proxy
19:09:40	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:09:40	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z	Started	Started container kserve-container
19:09:54	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:09:54	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz	Pulled	Successfully pulled image "docker.io/seldonio/mlserver:1.7.1" in 2m4.528s (2m4.528s including waiting). Image size: 10890461297 bytes.
19:09:54	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz	Started	Started container kserve-container
19:09:54	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz	Created	Created container: kube-rbac-proxy
19:09:54	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz	Started	Started container kube-rbac-proxy
19:09:54	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz	Created	Created container: kserve-container
19:10:14	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:10:14	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:10:19 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:10:19 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:10:29 (x10)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:10:29 (x10)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:10:31	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
19:10:41	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	InferenceServiceReady	InferenceService [sklearn-v2-mlserver] is Ready
19:10:41 (x11)	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:10:44	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz	Killing	Stopping container kube-rbac-proxy
19:10:44	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz	Killing	Stopping container kserve-container
19:10:49	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-runtime-predictor-9f86f76dd	SuccessfulCreate	Created pod: isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv
19:10:49	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-runtime-predictor-9f86f76dd from 0 to 1
19:10:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-runtime": the object has been modified; please apply your changes to the latest version and try again
19:10:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
19:10:50	kserve-ci-e2e-test	multus	isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv	AddedInterface	Add eth0 [10.132.0.41/23] from ovn-kubernetes
19:10:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv	Started	Started container storage-initializer
19:10:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv	Created	Created container: storage-initializer
19:10:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:10:51 (x2)	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz	Unhealthy	Readiness probe failed: Get "https://10.132.0.40:8643/healthz": dial tcp 10.132.0.40:8643: connect: connection refused
19:10:51	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz	Unhealthy	Readiness probe failed: Get "http://10.132.0.40:8080/v2/models/sklearn-v2-mlserver/ready": dial tcp 10.132.0.40:8080: connect: connection refused
19:11:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" already present on machine
19:11:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv	Started	Started container kube-rbac-proxy
19:11:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv	Created	Created container: kserve-container
19:11:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv	Started	Started container kserve-container
19:11:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:11:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv	Created	Created container: kube-rbac-proxy
19:11:10 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:11:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	InferenceServiceReady	InferenceService [isvc-autogluon-v2-noslash] is Ready
19:11:15 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv	Unhealthy	Readiness probe failed: dial tcp 10.132.0.41:8080: connect: connection refused
19:11:19 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:11:19 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:11:19 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:11:19 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:11:20	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z	Killing	Stopping container kserve-container
19:11:20	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-j2t5z	Killing	Stopping container kube-rbac-proxy
19:11:25	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-ts-v1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1": the object has been modified; please apply your changes to the latest version and try again
19:11:25	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-runtime] is Ready
19:11:25	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-ts-v1-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-ts-v1-predictor-6cf79d9f7d from 0 to 1
19:11:25	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-ts-v1-predictor-6cf79d9f7d	SuccessfulCreate	Created pod: isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p
19:11:26	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
19:11:26 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:11:27	kserve-ci-e2e-test	multus	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p	AddedInterface	Add eth0 [10.134.0.37/23] from ovn-kubernetes
19:11:27	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p	Started	Started container storage-initializer
19:11:27	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p	Created	Created container: storage-initializer
19:11:27	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:11:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv	Killing	Stopping container kserve-container
19:11:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv	Killing	Stopping container kube-rbac-proxy
19:11:33	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
19:11:33	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p	Started	Started container kserve-container
19:11:33	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p	Created	Created container: kserve-container
19:11:33	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p	Created	Created container: kube-rbac-proxy
19:11:33	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p	Started	Started container kube-rbac-proxy
19:11:33	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:11:37	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
19:11:37 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
19:11:37	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-runtime-predictor-5f69c7d498 from 0 to 1
19:11:37	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-runtime-predictor-5f69c7d498	SuccessfulCreate	Created pod: isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb
19:11:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb	Created	Created container: storage-initializer
19:11:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb	Started	Started container storage-initializer
19:11:38	kserve-ci-e2e-test	multus	isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb	AddedInterface	Add eth0 [10.132.0.42/23] from ovn-kubernetes
19:11:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:11:40	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:11:40	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:11:41 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p	Unhealthy	Readiness probe failed: dial tcp 10.134.0.37:8080: connect: connection refused
19:11:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb	Started	Started container kserve-container
19:11:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:11:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb	Created	Created container: kserve-container
19:11:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:11:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb	Created	Created container: kube-rbac-proxy
19:11:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb	Started	Started container kube-rbac-proxy
19:11:51 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:11:51	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	InferenceServiceReady	InferenceService [isvc-autogluon-ts-v1] is Ready
19:11:55	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:11:55	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:12:08	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p	Killing	Stopping container kserve-container
19:12:08	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-89n5p	Killing	Stopping container kube-rbac-proxy
19:12:13	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585	SuccessfulCreate	Created pod: isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg
19:12:13	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:12:13	kserve-ci-e2e-test	multus	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
19:12:13	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg	Created	Created container: storage-initializer
19:12:13	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-ts-v1-noslash-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-ts-v1-noslash-predictor-b54fbf585 from 0 to 1
19:12:13	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-ts-v1-noslash": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1-noslash": the object has been modified; please apply your changes to the latest version and try again
19:12:14	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg	Started	Started container storage-initializer
19:12:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
19:12:20	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg	Started	Started container kube-rbac-proxy
19:12:20	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg	Started	Started container kserve-container
19:12:20	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg	Created	Created container: kube-rbac-proxy
19:12:20	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:12:20	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg	Created	Created container: kserve-container
19:12:20	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
19:12:22 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:12:22 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:12:27 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg	Unhealthy	Readiness probe failed: dial tcp 10.134.0.38:8080: connect: connection refused
19:12:28	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:12:28	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:12:30	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-v2-runtime] is Ready
19:12:30 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:12:37	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:12:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb	Killing	Stopping container kube-rbac-proxy
19:12:37	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	InferenceServiceReady	InferenceService [isvc-autogluon-ts-v1-noslash] is Ready
19:12:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb	Killing	Stopping container kserve-container
19:12:37 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:12:37	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:12:37	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedUpdateStatus	horizontalpodautoscalers.autoscaling "isvc-sklearn-v2-runtime-predictor" not found
19:12:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb	Unhealthy	Readiness probe failed: Get "https://10.132.0.42:8643/healthz": dial tcp 10.132.0.42:8643: connect: connection refused
19:12:42	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-predictor-7b474946d	SuccessfulCreate	Created pod: isvc-sklearn-v2-predictor-7b474946d-56mbs
19:12:42	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-predictor-7b474946d from 0 to 1
19:12:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
19:12:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7b474946d-56mbs	Created	Created container: storage-initializer
19:12:43	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:12:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7b474946d-56mbs	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:12:43	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:12:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7b474946d-56mbs	Started	Started container storage-initializer
19:12:43	kserve-ci-e2e-test	multus	isvc-sklearn-v2-predictor-7b474946d-56mbs	AddedInterface	Add eth0 [10.132.0.43/23] from ovn-kubernetes
19:12:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7b474946d-56mbs	Started	Started container kube-rbac-proxy
19:12:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7b474946d-56mbs	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:12:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7b474946d-56mbs	Created	Created container: kube-rbac-proxy
19:12:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7b474946d-56mbs	Created	Created container: kserve-container
19:12:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7b474946d-56mbs	Started	Started container kserve-container
19:12:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7b474946d-56mbs	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" already present on machine
19:12:48	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg	Killing	Stopping container kube-rbac-proxy
19:12:48	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-skwgg	Killing	Stopping container kserve-container
19:12:53	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-lcngg	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-create-predictor-serving-cert" not found
19:12:53	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-qkb8c	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-create-predictor-serving-cert" not found
19:12:53	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-6cc94c6f4f	SuccessfulCreate	Created pod: isvc-canary-create-predictor-6cc94c6f4f-qkb8c
19:12:53	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-6cc94c6f4f	SuccessfulCreate	Created pod: isvc-canary-create-predictor-6cc94c6f4f-lcngg
19:12:53	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-create-v2-predictor-7855d67596 from 0 to 1
19:12:53	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-v2-predictor-7855d67596	SuccessfulCreate	Created pod: isvc-canary-create-v2-predictor-7855d67596-sgvxp
19:12:53	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-create-predictor-6cc94c6f4f from 0 to 2
19:12:53	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-7855d67596-sgvxp	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-create-v2-predictor-serving-cert" not found
19:12:54	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-7855d67596-sgvxp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:12:54	kserve-ci-e2e-test	multus	isvc-canary-create-predictor-6cc94c6f4f-qkb8c	AddedInterface	Add eth0 [10.132.0.45/23] from ovn-kubernetes
19:12:54	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-7855d67596-sgvxp	Created	Created container: storage-initializer
19:12:54	kserve-ci-e2e-test	multus	isvc-canary-create-predictor-6cc94c6f4f-lcngg	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
19:12:54	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-lcngg	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:12:54	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-qkb8c	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:12:54	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-qkb8c	Created	Created container: storage-initializer
19:12:54	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-7855d67596-sgvxp	Started	Started container storage-initializer
19:12:54	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-qkb8c	Started	Started container storage-initializer
19:12:54	kserve-ci-e2e-test	multus	isvc-canary-create-v2-predictor-7855d67596-sgvxp	AddedInterface	Add eth0 [10.132.0.44/23] from ovn-kubernetes
19:12:55	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-lcngg	Created	Created container: storage-initializer
19:12:55	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-lcngg	Started	Started container storage-initializer
19:12:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-qkb8c	Started	Started container kserve-container
19:12:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-qkb8c	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:12:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-7855d67596-sgvxp	Started	Started container kube-rbac-proxy
19:12:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-qkb8c	Created	Created container: kserve-container
19:12:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-qkb8c	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" already present on machine
19:12:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-7855d67596-sgvxp	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" already present on machine
19:12:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-7855d67596-sgvxp	Created	Created container: kserve-container
19:12:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-7855d67596-sgvxp	Created	Created container: kube-rbac-proxy
19:12:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-7855d67596-sgvxp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:12:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-qkb8c	Created	Created container: kube-rbac-proxy
19:12:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-qkb8c	Started	Started container kube-rbac-proxy
19:12:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-7855d67596-sgvxp	Started	Started container kserve-container
19:12:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-lcngg	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986"
19:13:12	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-lcngg	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:13:12	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-lcngg	Started	Started container kube-rbac-proxy
19:13:12	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-lcngg	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" in 13.336s (13.336s including waiting). Image size: 1580859293 bytes.
19:13:12	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-lcngg	Created	Created container: kserve-container
19:13:12	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-lcngg	Started	Started container kserve-container
19:13:12 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:13:12 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:13:12	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-lcngg	Created	Created container: kube-rbac-proxy
19:14:05 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:14:05	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	InferenceServiceReady	InferenceService [isvc-sklearn-v2] is Ready
19:14:07 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-qkb8c	Unhealthy	Readiness probe failed: dial tcp 10.132.0.45:8080: connect: connection refused
19:14:12 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:14:12 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:14:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7b474946d-56mbs	Killing	Stopping container kube-rbac-proxy
19:14:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7b474946d-56mbs	Killing	Stopping container kserve-container
19:14:15 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7b474946d-56mbs	Unhealthy	Readiness probe failed: dial tcp 10.132.0.43:8080: connect: connection refused
19:14:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-7b474946d-56mbs	Unhealthy	Readiness probe failed: Get "https://10.132.0.43:8643/healthz": dial tcp 10.132.0.43:8643: connect: connection refused
19:14:16	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-6cc94c6f4f	SuccessfulDelete	Deleted pod: isvc-canary-create-predictor-6cc94c6f4f-qkb8c
19:14:16	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-qkb8c	Killing	Stopping container kube-rbac-proxy
19:14:16	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-create-predictor-6cc94c6f4f from 2 to 1
19:14:16	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-qkb8c	Killing	Stopping container kserve-container
19:14:16 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	UpdateFailed	Failed to update status for InferenceService "isvc-canary-create": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again
19:14:16 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again
19:14:17	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-qkb8c	Unhealthy	Readiness probe failed: Get "https://10.132.0.45:8643/healthz": dial tcp 10.132.0.45:8643: connect: connection refused
19:14:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
19:14:18	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-mixed-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-mixed-predictor-8cf94477c from 0 to 1
19:14:18	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-mixed-predictor-8cf94477c	SuccessfulCreate	Created pod: isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh
19:14:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-mixed": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-mixed": the object has been modified; please apply your changes to the latest version and try again
19:14:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:14:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh	Started	Started container storage-initializer
19:14:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh	Created	Created container: storage-initializer
19:14:19	kserve-ci-e2e-test	multus	isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
19:14:20 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-lcngg	Unhealthy	Readiness probe failed: dial tcp 10.134.0.39:8080: connect: connection refused
19:14:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh	Started	Started container kserve-container
19:14:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh	Created	Created container: kserve-container
19:14:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" already present on machine
19:14:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:14:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh	Created	Created container: kube-rbac-proxy
19:14:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh	Started	Started container kube-rbac-proxy
19:14:30	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	InferenceServiceReady	InferenceService [isvc-canary-create] is Ready
19:14:30 (x21)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:14:33	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-7855d67596-sgvxp	Killing	Stopping container kube-rbac-proxy
19:14:33	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:14:33	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:14:33	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-7855d67596-sgvxp	Killing	Stopping container kserve-container
19:14:33	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-lcngg	Killing	Stopping container kube-rbac-proxy
19:14:33	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-lcngg	Killing	Stopping container kserve-container
19:14:35	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6cc94c6f4f-lcngg	Unhealthy	Readiness probe failed: Get "https://10.134.0.39:8643/healthz": dial tcp 10.134.0.39:8643: connect: connection refused
19:14:36	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-7855d67596-sgvxp	Unhealthy	Readiness probe failed: Get "https://10.132.0.44:8643/healthz": dial tcp 10.132.0.44:8643: connect: connection refused
19:14:36 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-7855d67596-sgvxp	Unhealthy	Readiness probe failed: dial tcp 10.132.0.44:8080: connect: connection refused
19:14:38	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-predictor-b9b98bcd9 from 0 to 2
19:14:38	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-b9b98bcd9	SuccessfulCreate	Created pod: isvc-canary-promote-predictor-b9b98bcd9-bj7dj
19:14:38	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-v2-predictor-7cc6db9b5d	SuccessfulCreate	Created pod: isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m
19:14:38	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-v2-predictor-7cc6db9b5d from 0 to 1
19:14:39	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-bj7dj	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-promote-predictor-serving-cert" not found
19:14:39	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
19:14:39	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-b9b98bcd9	SuccessfulCreate	Created pod: isvc-canary-promote-predictor-b9b98bcd9-tdnsw
19:14:39	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-tdnsw	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-promote-predictor-serving-cert" not found
19:14:40	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-tdnsw	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:14:40	kserve-ci-e2e-test	multus	isvc-canary-promote-predictor-b9b98bcd9-tdnsw	AddedInterface	Add eth0 [10.134.0.42/23] from ovn-kubernetes
19:14:40	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-tdnsw	Created	Created container: storage-initializer
19:14:40	kserve-ci-e2e-test	multus	isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m	AddedInterface	Add eth0 [10.134.0.41/23] from ovn-kubernetes
19:14:40	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:14:40	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m	Created	Created container: storage-initializer
19:14:40	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m	Started	Started container storage-initializer
19:14:40	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-bj7dj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:14:40	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-tdnsw	Started	Started container storage-initializer
19:14:40	kserve-ci-e2e-test	multus	isvc-canary-promote-predictor-b9b98bcd9-bj7dj	AddedInterface	Add eth0 [10.132.0.46/23] from ovn-kubernetes
19:14:40	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-bj7dj	Started	Started container storage-initializer
19:14:40	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-bj7dj	Created	Created container: storage-initializer
19:14:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m	Created	Created container: kserve-container
19:14:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:14:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-bj7dj	Created	Created container: kserve-container
19:14:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-bj7dj	Started	Started container kserve-container
19:14:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-bj7dj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:14:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-bj7dj	Created	Created container: kube-rbac-proxy
19:14:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-bj7dj	Started	Started container kube-rbac-proxy
19:14:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m	Started	Started container kube-rbac-proxy
19:14:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m	Created	Created container: kube-rbac-proxy
19:14:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-bj7dj	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" already present on machine
19:14:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m	Started	Started container kserve-container
19:14:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-tdnsw	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" already present on machine
19:14:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-tdnsw	Created	Created container: kserve-container
19:14:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-tdnsw	Started	Started container kserve-container
19:14:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-tdnsw	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:14:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-tdnsw	Created	Created container: kube-rbac-proxy
19:14:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-tdnsw	Started	Started container kube-rbac-proxy
19:14:44	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" already present on machine
19:15:41 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:15:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	InferenceServiceReady	InferenceService [isvc-sklearn-v2-mixed] is Ready
19:15:48 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:15:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh	Killing	Stopping container kserve-container
19:15:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh	Killing	Stopping container kube-rbac-proxy
19:15:48 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:15:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh	Unhealthy	Readiness probe failed: Get "https://10.134.0.40:8643/healthz": dial tcp 10.134.0.40:8643: connect: connection refused
19:15:51 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-8cf94477c-h4bmh	Unhealthy	Readiness probe failed: dial tcp 10.134.0.40:8080: connect: connection refused
19:15:52 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-bj7dj	Unhealthy	Readiness probe failed: dial tcp 10.132.0.46:8080: connect: connection refused
19:15:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow": the object has been modified; please apply your changes to the latest version and try again
19:15:53	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-ssm8l	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-tensorflow-predictor-serving-cert" not found
19:15:53	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-predictor-547dfc5669	SuccessfulCreate	Created pod: isvc-tensorflow-predictor-547dfc5669-ssm8l
19:15:53	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-predictor-547dfc5669 from 0 to 1
19:15:54	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-ssm8l	Started	Started container storage-initializer
19:15:54	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-ssm8l	Created	Created container: storage-initializer
19:15:54	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-ssm8l	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:15:54	kserve-ci-e2e-test	multus	isvc-tensorflow-predictor-547dfc5669-ssm8l	AddedInterface	Add eth0 [10.132.0.47/23] from ovn-kubernetes
19:15:59	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-ssm8l	Pulling	Pulling image "tensorflow/serving:2.6.2"
19:16:01	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-promote-predictor-b9b98bcd9 from 2 to 1
19:16:01 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again
19:16:01 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	UpdateFailed	Failed to update status for InferenceService "isvc-canary-promote": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again
19:16:01	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-b9b98bcd9	SuccessfulDelete	Deleted pod: isvc-canary-promote-predictor-b9b98bcd9-bj7dj
19:16:01	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-bj7dj	Killing	Stopping container kserve-container
19:16:01	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-bj7dj	Killing	Stopping container kube-rbac-proxy
19:16:02	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-bj7dj	Unhealthy	Readiness probe failed: Get "https://10.132.0.46:8643/healthz": dial tcp 10.132.0.46:8643: connect: connection refused
19:16:02	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-ssm8l	Created	Created container: kserve-container
19:16:02	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-ssm8l	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 3.602s (3.602s including waiting). Image size: 425873876 bytes.
19:16:02	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-ssm8l	Started	Started container kserve-container
19:16:02	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-ssm8l	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:16:02	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-ssm8l	Created	Created container: kube-rbac-proxy
19:16:02	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-ssm8l	Started	Started container kube-rbac-proxy
19:16:08 (x21)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:16:08	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-v2-predictor-7cc6db9b5d from 1 to 2
19:16:09	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-tdnsw	Killing	Stopping container kube-rbac-proxy
19:16:09 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-ssm8l	Unhealthy	Readiness probe failed: dial tcp 10.132.0.47:8080: connect: connection refused
19:16:09	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-tdnsw	Killing	Stopping container kserve-container
19:16:09	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-v2-predictor-7cc6db9b5d	SuccessfulCreate	Created pod: isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x
19:16:09	kserve-ci-e2e-test	multus	isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x	AddedInterface	Add eth0 [10.132.0.48/23] from ovn-kubernetes
19:16:09	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:16:09	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x	Created	Created container: storage-initializer
19:16:09	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x	Started	Started container storage-initializer
19:16:12 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-tdnsw	Unhealthy	Readiness probe failed: dial tcp 10.134.0.42:8080: connect: connection refused
19:16:12	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-b9b98bcd9-tdnsw	Unhealthy	Readiness probe failed: Get "https://10.134.0.42:8643/healthz": dial tcp 10.134.0.42:8643: connect: connection refused
19:16:14	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x	Started	Started container kube-rbac-proxy
19:16:14	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" already present on machine
19:16:14	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x	Created	Created container: kserve-container
19:16:14	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x	Started	Started container kserve-container
19:16:14	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:16:14	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x	Created	Created container: kube-rbac-proxy
19:16:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	InferenceServiceReady	InferenceService [isvc-tensorflow] is Ready
19:16:19 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:16:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:16:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:16:34	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-ssm8l	Killing	Stopping container kube-rbac-proxy
19:16:34	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-ssm8l	Killing	Stopping container kserve-container
19:16:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow-runtime": the object has been modified; please apply your changes to the latest version and try again
19:16:39	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-runtime-predictor-855cfcd677 from 0 to 1
19:16:40	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
19:16:40	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-runtime-predictor-855cfcd677	SuccessfulCreate	Created pod: isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2
19:16:41	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2	Created	Created container: storage-initializer
19:16:41	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:16:41	kserve-ci-e2e-test	multus	isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2	AddedInterface	Add eth0 [10.134.0.43/23] from ovn-kubernetes
19:16:41	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2	Started	Started container storage-initializer
19:16:46	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2	Pulling	Pulling image "tensorflow/serving:2.6.2"
19:16:50	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2	Started	Started container kube-rbac-proxy
19:16:50	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2	Created	Created container: kube-rbac-proxy
19:16:50	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:16:50	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2	Started	Started container kserve-container
19:16:50	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2	Created	Created container: kserve-container
19:16:50	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 3.636s (3.636s including waiting). Image size: 425873876 bytes.
19:16:54	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:16:54	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:16:56 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2	Unhealthy	Readiness probe failed: dial tcp 10.134.0.43:8080: connect: connection refused
19:17:04 (x6)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-ssm8l	Unhealthy	Readiness probe failed: Get "https://10.132.0.47:8643/healthz": dial tcp 10.132.0.47:8643: connect: connection refused
19:17:06	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	InferenceServiceReady	InferenceService [isvc-tensorflow-runtime] is Ready
19:17:06 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:17:10	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:17:10	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:17:21	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2	Killing	Stopping container kube-rbac-proxy
19:17:21	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2	Killing	Stopping container kserve-container
19:17:26	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	UpdateFailed	Failed to update status for InferenceService "isvc-triton": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-triton": the object has been modified; please apply your changes to the latest version and try again
19:17:26	kserve-ci-e2e-test	replicaset-controller	isvc-triton-predictor-5d7db58676	SuccessfulCreate	Created pod: isvc-triton-predictor-5d7db58676-gpgpq
19:17:26	kserve-ci-e2e-test	deployment-controller	isvc-triton-predictor	ScalingReplicaSet	Scaled up replica set isvc-triton-predictor-5d7db58676 from 0 to 1
19:17:26	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-gpgpq	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-triton-predictor-serving-cert" not found
19:17:27	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-gpgpq	Created	Created container: storage-initializer
19:17:27	kserve-ci-e2e-test	multus	isvc-triton-predictor-5d7db58676-gpgpq	AddedInterface	Add eth0 [10.132.0.49/23] from ovn-kubernetes
19:17:27	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-gpgpq	Started	Started container storage-initializer
19:17:27	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-gpgpq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:17:31	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-gpgpq	Pulling	Pulling image "nvcr.io/nvidia/tritonserver:23.05-py3"
19:17:32 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	InferenceServiceReady	InferenceService [isvc-canary-promote] is Ready
19:17:39	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m	Killing	Stopping container kserve-container
19:17:39	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m	Killing	Stopping container kube-rbac-proxy
19:17:39	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x	Killing	Stopping container kserve-container
19:17:39	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x	Killing	Stopping container kube-rbac-proxy
19:17:41	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m	Unhealthy	Readiness probe failed: Get "https://10.134.0.41:8643/healthz": dial tcp 10.134.0.41:8643: connect: connection refused
19:17:41 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-zql6m	Unhealthy	Readiness probe failed: dial tcp 10.134.0.41:8080: connect: connection refused
19:17:42 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x	Unhealthy	Readiness probe failed: dial tcp 10.132.0.48:8080: connect: connection refused
19:17:42	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x	Unhealthy	Readiness probe failed: Get "https://10.132.0.48:8643/healthz": dial tcp 10.132.0.48:8643: connect: connection refused
19:17:44	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-v2-predictor-7dc597df9b	SuccessfulCreate	Created pod: isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt
19:17:44	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-v2-predictor-7dc597df9b from 0 to 1
19:17:44	kserve-ci-e2e-test	multus	isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt	AddedInterface	Add eth0 [10.134.0.44/23] from ovn-kubernetes
19:17:44	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-predictor-665c868b97 from 0 to 2
19:17:44	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-665c868b97	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-665c868b97-vqw4s
19:17:44	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-665c868b97	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-665c868b97-8fg87
19:17:44	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-8fg87	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-predictor-serving-cert" not found
19:17:45	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:17:45	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-665c868b97-8fg87	AddedInterface	Add eth0 [10.134.0.45/23] from ovn-kubernetes
19:17:45	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt	Started	Started container storage-initializer
19:17:45	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt	Created	Created container: storage-initializer
19:17:45	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-vqw4s	Created	Created container: storage-initializer
19:17:45	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-vqw4s	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:17:45	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-8fg87	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:17:45	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-8fg87	Created	Created container: storage-initializer
19:17:45	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-8fg87	Started	Started container storage-initializer
19:17:45	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-vqw4s	Started	Started container storage-initializer
19:17:45	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-665c868b97-vqw4s	AddedInterface	Add eth0 [10.132.0.50/23] from ovn-kubernetes
19:17:48	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt	Created	Created container: kube-rbac-proxy
19:17:48	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt	Started	Started container kserve-container
19:17:48	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" already present on machine
19:17:48	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:17:48	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt	Created	Created container: kserve-container
19:17:48	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt	Started	Started container kube-rbac-proxy
19:17:49	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-8fg87	Started	Started container kserve-container
19:17:49	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-8fg87	Created	Created container: kube-rbac-proxy
19:17:49	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-8fg87	Created	Created container: kserve-container
19:17:49	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-8fg87	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:17:49	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-8fg87	Started	Started container kube-rbac-proxy
19:17:49	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-8fg87	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" already present on machine
19:17:50	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-vqw4s	Created	Created container: kserve-container
19:17:50	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-vqw4s	Started	Started container kserve-container
19:17:50	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-vqw4s	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:17:50	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-vqw4s	Created	Created container: kube-rbac-proxy
19:17:50	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-vqw4s	Started	Started container kube-rbac-proxy
19:17:50	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-vqw4s	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" already present on machine
19:17:51 (x7)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-9mvh2	Unhealthy	Readiness probe failed: Get "https://10.134.0.43:8643/healthz": dial tcp 10.134.0.43:8643: connect: connection refused
19:18:56 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-8fg87	Unhealthy	Readiness probe failed: dial tcp 10.134.0.45:8080: connect: connection refused
19:19:05	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-rollback-predictor-665c868b97 from 2 to 1
19:19:05	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-665c868b97	SuccessfulDelete	Deleted pod: isvc-canary-rollback-predictor-665c868b97-8fg87
19:19:05	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-8fg87	Killing	Stopping container kube-rbac-proxy
19:19:05	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-8fg87	Killing	Stopping container kserve-container
19:19:05 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again
19:19:05 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	UpdateFailed	Failed to update status for InferenceService "isvc-canary-rollback": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again
19:19:06	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-8fg87	Unhealthy	Readiness probe failed: Get "https://10.134.0.45:8643/healthz": dial tcp 10.134.0.45:8643: connect: connection refused
19:19:30	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-gpgpq	Created	Created container: kserve-container
19:19:30	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-gpgpq	Pulled	Successfully pulled image "nvcr.io/nvidia/tritonserver:23.05-py3" in 1m58.544s (1m58.544s including waiting). Image size: 12907074623 bytes.
19:19:30	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-gpgpq	Created	Created container: kube-rbac-proxy
19:19:30	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-gpgpq	Started	Started container kube-rbac-proxy
19:19:30	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-gpgpq	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:19:30	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-gpgpq	Started	Started container kserve-container
19:19:33 (x2)	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-gpgpq	Unhealthy	Readiness probe failed: dial tcp 10.132.0.49:8080: connect: connection refused
19:19:34	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt	Killing	Stopping container kserve-container
19:19:34	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt	Killing	Stopping container kube-rbac-proxy
19:19:34 (x21)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:19:34	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-predictor-665c868b97 from 1 to 2
19:19:34	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-665c868b97	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-665c868b97-tm5sv
19:19:35	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt	Unhealthy	Readiness probe failed: Get "https://10.134.0.44:8643/healthz": dial tcp 10.134.0.44:8643: connect: connection refused
19:19:35	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-tm5sv	Created	Created container: storage-initializer
19:19:35 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7dc597df9b-tgwpt	Unhealthy	Readiness probe failed: dial tcp 10.134.0.44:8080: connect: connection refused
19:19:35	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-665c868b97-tm5sv	AddedInterface	Add eth0 [10.134.0.46/23] from ovn-kubernetes
19:19:35	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-tm5sv	Started	Started container storage-initializer
19:19:35	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-tm5sv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:19:38 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:19:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	InferenceServiceReady	InferenceService [isvc-triton] is Ready
19:19:39	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-tm5sv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:19:39	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-tm5sv	Started	Started container kube-rbac-proxy
19:19:39	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-tm5sv	Created	Created container: kube-rbac-proxy
19:19:39	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-tm5sv	Started	Started container kserve-container
19:19:39	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-tm5sv	Created	Created container: kserve-container
19:19:39	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-tm5sv	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" already present on machine
19:19:41 (x9)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:19:41 (x9)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:19:48	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-gpgpq	Killing	Stopping container kube-rbac-proxy
19:19:48	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-gpgpq	Killing	Stopping container kserve-container
19:19:53	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-predictor-7566d65bf7	SuccessfulCreate	Created pod: isvc-xgboost-predictor-7566d65bf7-ccl6q
19:19:53	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-predictor-7566d65bf7 from 0 to 1
19:19:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost": the object has been modified; please apply your changes to the latest version and try again
19:19:54	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ccl6q	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:19:54	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ccl6q	Created	Created container: storage-initializer
19:19:54	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ccl6q	Started	Started container storage-initializer
19:19:54	kserve-ci-e2e-test	multus	isvc-xgboost-predictor-7566d65bf7-ccl6q	AddedInterface	Add eth0 [10.134.0.47/23] from ovn-kubernetes
19:19:58	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ccl6q	Pulling	Pulling image "kserve/xgbserver:latest"
19:20:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ccl6q	Started	Started container kserve-container
19:20:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ccl6q	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:20:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ccl6q	Created	Created container: kube-rbac-proxy
19:20:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ccl6q	Started	Started container kube-rbac-proxy
19:20:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ccl6q	Created	Created container: kserve-container
19:20:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ccl6q	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 17.191s (17.191s including waiting). Image size: 1332365570 bytes.
19:20:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:20:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:21:05 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	InferenceServiceReady	InferenceService [isvc-canary-rollback] is Ready
19:21:13 (x8)	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ccl6q	Unhealthy	Readiness probe failed: dial tcp 10.134.0.47:8080: connect: connection refused
19:21:14	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-tm5sv	Killing	Stopping container kserve-container
19:21:14	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-vqw4s	Killing	Stopping container kserve-container
19:21:14	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-tm5sv	Killing	Stopping container kube-rbac-proxy
19:21:14	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-vqw4s	Killing	Stopping container kube-rbac-proxy
19:21:15 (x11)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-tm5sv	Unhealthy	Readiness probe failed: dial tcp 10.134.0.46:8080: connect: connection refused
19:21:15 (x12)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-vqw4s	Unhealthy	Readiness probe failed: dial tcp 10.132.0.50:8080: connect: connection refused
19:21:17	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-tm5sv	Unhealthy	Readiness probe failed: Get "https://10.134.0.46:8643/healthz": dial tcp 10.134.0.46:8643: connect: connection refused
19:21:17	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-665c868b97-vqw4s	Unhealthy	Readiness probe failed: Get "https://10.132.0.50:8643/healthz": dial tcp 10.132.0.50:8643: connect: connection refused
19:21:19	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-stop-predictor-7bb9c548b5 from 0 to 2
19:21:19	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-7bb9c548b5	SuccessfulCreate	Created pod: isvc-canary-stop-predictor-7bb9c548b5-xr4pd
19:21:19	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-stop-v2-predictor-b7c9ff648 from 0 to 1
19:21:19	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-v2-predictor-b7c9ff648	SuccessfulCreate	Created pod: isvc-canary-stop-v2-predictor-b7c9ff648-65lhc
19:21:20	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-48h5k	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-stop-predictor-serving-cert" not found
19:21:20	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-xr4pd	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-stop-predictor-serving-cert" not found
19:21:20	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-b7c9ff648-65lhc	Created	Created container: storage-initializer
19:21:20	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-b7c9ff648-65lhc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:21:20	kserve-ci-e2e-test	multus	isvc-canary-stop-v2-predictor-b7c9ff648-65lhc	AddedInterface	Add eth0 [10.134.0.48/23] from ovn-kubernetes
19:21:20	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-b7c9ff648-65lhc	Started	Started container storage-initializer
19:21:20	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-7bb9c548b5	SuccessfulCreate	Created pod: isvc-canary-stop-predictor-7bb9c548b5-48h5k
19:21:21	kserve-ci-e2e-test	multus	isvc-canary-stop-predictor-7bb9c548b5-48h5k	AddedInterface	Add eth0 [10.134.0.49/23] from ovn-kubernetes
19:21:21	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-xr4pd	Started	Started container storage-initializer
19:21:21	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-xr4pd	Created	Created container: storage-initializer
19:21:21	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-xr4pd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:21:21	kserve-ci-e2e-test	multus	isvc-canary-stop-predictor-7bb9c548b5-xr4pd	AddedInterface	Add eth0 [10.132.0.51/23] from ovn-kubernetes
19:21:21	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-48h5k	Started	Started container storage-initializer
19:21:21	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-48h5k	Created	Created container: storage-initializer
19:21:21	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-48h5k	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:21:23	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	InferenceServiceReady	InferenceService [isvc-xgboost] is Ready
19:21:23 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:21:23 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:21:23 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:21:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ccl6q	Killing	Stopping container kube-rbac-proxy
19:21:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-ccl6q	Killing	Stopping container kserve-container
19:21:25	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-xr4pd	Started	Started container kube-rbac-proxy
19:21:25	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-xr4pd	Created	Created container: kube-rbac-proxy
19:21:25	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-b7c9ff648-65lhc	Created	Created container: kserve-container
19:21:25	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-b7c9ff648-65lhc	Started	Started container kserve-container
19:21:25	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-48h5k	Created	Created container: kserve-container
19:21:25	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-b7c9ff648-65lhc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:21:25	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-b7c9ff648-65lhc	Created	Created container: kube-rbac-proxy
19:21:25	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-b7c9ff648-65lhc	Started	Started container kube-rbac-proxy
19:21:25	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-48h5k	Started	Started container kserve-container
19:21:25	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-48h5k	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:21:25	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-48h5k	Created	Created container: kube-rbac-proxy
19:21:25	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-48h5k	Started	Started container kube-rbac-proxy
19:21:25	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-b7c9ff648-65lhc	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" already present on machine
19:21:25	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-48h5k	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" already present on machine
19:21:25	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-xr4pd	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" already present on machine
19:21:25	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-xr4pd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:21:25	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-xr4pd	Started	Started container kserve-container
19:21:25	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-xr4pd	Created	Created container: kserve-container
19:21:28	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-mlserver-predictor-7b55b84f54 from 0 to 1
19:21:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
19:21:29	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-mlserver-predictor-7b55b84f54	SuccessfulCreate	Created pod: isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj
19:21:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj	Started	Started container storage-initializer
19:21:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj	Created	Created container: storage-initializer
19:21:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:21:29	kserve-ci-e2e-test	multus	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj	AddedInterface	Add eth0 [10.132.0.52/23] from ovn-kubernetes
19:21:33	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj	Created	Created container: kube-rbac-proxy
19:21:33	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj	Started	Started container kube-rbac-proxy
19:21:33	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:21:33	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj	Created	Created container: kserve-container
19:21:33	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj	Started	Started container kserve-container
19:21:33	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:21:59 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:21:59 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:22:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	InferenceServiceReady	InferenceService [isvc-xgboost-v2-mlserver] is Ready
19:22:11 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:22:14	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:22:14	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:22:19	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj	Killing	Stopping container kube-rbac-proxy
19:22:19	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj	Killing	Stopping container kserve-container
19:22:21	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj	Unhealthy	Readiness probe failed: Get "https://10.132.0.52:8643/healthz": dial tcp 10.132.0.52:8643: connect: connection refused
19:22:21	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj	Unhealthy	Readiness probe failed: Get "http://10.132.0.52:8080/v2/models/isvc-xgboost-v2-mlserver/ready": dial tcp 10.132.0.52:8080: connect: connection refused
19:22:24	kserve-ci-e2e-test	deployment-controller	xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set xgboost-v2-mlserver-predictor-59944bcf4c from 0 to 1
19:22:24	kserve-ci-e2e-test	replicaset-controller	xgboost-v2-mlserver-predictor-59944bcf4c	SuccessfulCreate	Created pod: xgboost-v2-mlserver-predictor-59944bcf4c-d5prm
19:22:24	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-d5prm	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "xgboost-v2-mlserver-predictor-serving-cert" not found
19:22:24	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
19:22:25	kserve-ci-e2e-test	multus	xgboost-v2-mlserver-predictor-59944bcf4c-d5prm	AddedInterface	Add eth0 [10.132.0.53/23] from ovn-kubernetes
19:22:25	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-d5prm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:22:25	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-d5prm	Started	Started container storage-initializer
19:22:25	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-d5prm	Created	Created container: storage-initializer
19:22:29	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-d5prm	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:22:30	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-d5prm	Created	Created container: kserve-container
19:22:30	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-d5prm	Started	Started container kserve-container
19:22:30	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-d5prm	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:22:30	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-d5prm	Created	Created container: kube-rbac-proxy
19:22:30	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-d5prm	Started	Started container kube-rbac-proxy
19:22:39	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:22:39 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-b7c9ff648-65lhc	Unhealthy	Readiness probe failed: dial tcp 10.134.0.48:8080: connect: connection refused
19:22:39	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:22:43	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	InferenceServiceReady	InferenceService [isvc-canary-stop] is Ready
19:22:43 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again
19:22:49	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-stop-predictor-7bb9c548b5 from 2 to 1
19:22:49	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-7bb9c548b5	SuccessfulDelete	Deleted pod: isvc-canary-stop-predictor-7bb9c548b5-48h5k
19:22:49 (x20)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:22:49 (x3)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	UpdateFailed	Failed to update status for InferenceService "isvc-canary-stop": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again
19:22:49	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-xr4pd	Killing	Stopping container kube-rbac-proxy
19:22:49	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-48h5k	Killing	Stopping container kube-rbac-proxy
19:22:49	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-48h5k	Killing	Stopping container kserve-container
19:22:49	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-xr4pd	Killing	Stopping container kserve-container
19:22:50	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-b7c9ff648-65lhc	Killing	Stopping container kube-rbac-proxy
19:22:50	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-b7c9ff648-65lhc	Killing	Stopping container kserve-container
19:22:52	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-b7c9ff648-65lhc	Unhealthy	Readiness probe failed: Get "https://10.134.0.48:8643/healthz": dial tcp 10.134.0.48:8643: connect: connection refused
19:22:53 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-xr4pd	Unhealthy	Readiness probe failed: dial tcp 10.132.0.51:8080: connect: connection refused
19:22:53	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-48h5k	Unhealthy	Readiness probe failed: Get "https://10.134.0.49:8643/healthz": dial tcp 10.134.0.49:8643: connect: connection refused
19:22:53	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-xr4pd	Unhealthy	Readiness probe failed: Get "https://10.132.0.51:8643/healthz": dial tcp 10.132.0.51:8643: connect: connection refused
19:22:53 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-7bb9c548b5-48h5k	Unhealthy	Readiness probe failed: dial tcp 10.134.0.49:8080: connect: connection refused
19:23:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-fvgb2	Started	Started container storage-initializer
19:23:00	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-predictor-54f85ddf88 from 0 to 1
19:23:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-fvgb2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:23:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-fvgb2	Created	Created container: storage-initializer
19:23:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm": the object has been modified; please apply your changes to the latest version and try again
19:23:00	kserve-ci-e2e-test	multus	isvc-lightgbm-predictor-54f85ddf88-fvgb2	AddedInterface	Add eth0 [10.134.0.50/23] from ovn-kubernetes
19:23:00	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-predictor-54f85ddf88	SuccessfulCreate	Created pod: isvc-lightgbm-predictor-54f85ddf88-fvgb2
19:23:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-fvgb2	Pulling	Pulling image "kserve/lgbserver:latest"
19:23:08 (x11)	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:23:08	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	InferenceServiceReady	InferenceService [xgboost-v2-mlserver] is Ready
19:23:09 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:23:09 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:23:11	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-fvgb2	Created	Created container: kube-rbac-proxy
19:23:11	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-fvgb2	Started	Started container kube-rbac-proxy
19:23:11	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-fvgb2	Pulled	Successfully pulled image "kserve/lgbserver:latest" in 6.084s (6.084s including waiting). Image size: 632192787 bytes.
19:23:11	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-fvgb2	Created	Created container: kserve-container
19:23:11	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-fvgb2	Started	Started container kserve-container
19:23:11	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-fvgb2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:23:14	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-d5prm	Killing	Stopping container kube-rbac-proxy
19:23:14	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-d5prm	Killing	Stopping container kserve-container
19:23:17	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-d5prm	Unhealthy	Readiness probe failed: Get "https://10.132.0.53:8643/healthz": dial tcp 10.132.0.53:8643: connect: connection refused
19:23:18	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-d5prm	Unhealthy	Readiness probe failed: Get "http://10.132.0.53:8080/v2/models/xgboost-v2-mlserver/ready": dial tcp 10.132.0.53:8080: connect: connection refused
19:23:19	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-runtime-predictor-54dd488c85	SuccessfulCreate	Created pod: isvc-xgboost-runtime-predictor-54dd488c85-l49vq
19:23:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-runtime": the object has been modified; please apply your changes to the latest version and try again
19:23:19	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-l49vq	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
19:23:19	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-runtime-predictor-54dd488c85 from 0 to 1
19:23:20	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-l49vq	Started	Started container storage-initializer
19:23:20	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-l49vq	Created	Created container: storage-initializer
19:23:20	kserve-ci-e2e-test	multus	isvc-xgboost-runtime-predictor-54dd488c85-l49vq	AddedInterface	Add eth0 [10.134.0.51/23] from ovn-kubernetes
19:23:20	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-l49vq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:23:25	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-l49vq	Created	Created container: kube-rbac-proxy
19:23:25	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-l49vq	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:23:25	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-l49vq	Started	Started container kube-rbac-proxy
19:23:25	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-l49vq	Started	Started container kserve-container
19:23:25	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-l49vq	Created	Created container: kserve-container
19:23:25	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-l49vq	Pulled	Container image "kserve/xgbserver:latest" already present on machine
19:23:45 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:23:45 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:24:04 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:24:04 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:24:19 (x9)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-fvgb2	Unhealthy	Readiness probe failed: dial tcp 10.134.0.50:8080: connect: connection refused
19:24:23 (x8)	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-l49vq	Unhealthy	Readiness probe failed: dial tcp 10.134.0.51:8080: connect: connection refused
19:24:29	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	InferenceServiceReady	InferenceService [isvc-lightgbm] is Ready
19:24:29 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:24:30 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:24:30 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:24:30	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-fvgb2	Killing	Stopping container kserve-container
19:24:30	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-fvgb2	Killing	Stopping container kube-rbac-proxy
19:24:33	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-runtime] is Ready
19:24:33 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:24:34 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:24:34	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-fvgb2	Unhealthy	Readiness probe failed: Get "https://10.134.0.50:8643/healthz": dial tcp 10.134.0.50:8643: connect: connection refused
19:24:34 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:24:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-runtime": the object has been modified; please apply your changes to the latest version and try again
19:24:35	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-runtime-predictor-577f7db7cb from 0 to 1
19:24:35	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-888mm	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
19:24:35	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-runtime-predictor-577f7db7cb	SuccessfulCreate	Created pod: isvc-lightgbm-runtime-predictor-577f7db7cb-888mm
19:24:36	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-888mm	Created	Created container: storage-initializer
19:24:36	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-888mm	Started	Started container storage-initializer
19:24:36	kserve-ci-e2e-test	multus	isvc-lightgbm-runtime-predictor-577f7db7cb-888mm	AddedInterface	Add eth0 [10.132.0.54/23] from ovn-kubernetes
19:24:36	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-888mm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:24:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-l49vq	Killing	Stopping container kserve-container
19:24:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-l49vq	Killing	Stopping container kube-rbac-proxy
19:24:40	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-888mm	Pulling	Pulling image "kserve/lgbserver:latest"
19:24:44	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-runtime-predictor-7787c55dc9	SuccessfulCreate	Created pod: isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b
19:24:44	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-runtime-predictor-7787c55dc9 from 0 to 1
19:24:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
19:24:44 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
19:24:46	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-888mm	Created	Created container: kube-rbac-proxy
19:24:46	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-888mm	Started	Started container kserve-container
19:24:46	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-888mm	Started	Started container kube-rbac-proxy
19:24:46	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-888mm	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:24:46	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-888mm	Pulled	Successfully pulled image "kserve/lgbserver:latest" in 6.158s (6.158s including waiting). Image size: 632192787 bytes.
19:24:46	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-888mm	Created	Created container: kserve-container
19:24:46	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b	Started	Started container storage-initializer
19:24:46	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b	Created	Created container: storage-initializer
19:24:46	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:24:46	kserve-ci-e2e-test	multus	isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b	AddedInterface	Add eth0 [10.132.0.55/23] from ovn-kubernetes
19:24:51	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:24:51	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b	Started	Started container kserve-container
19:24:51	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b	Created	Created container: kserve-container
19:24:51	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:24:51	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b	Created	Created container: kube-rbac-proxy
19:24:51	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b	Started	Started container kube-rbac-proxy
19:25:20 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:25:20 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:25:28	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
19:25:29 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:25:29 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:25:38 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:25:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-v2-runtime] is Ready
19:25:44	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:25:44	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:25:44	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b	Killing	Stopping container kube-rbac-proxy
19:25:44	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b	Killing	Stopping container kserve-container
19:25:48	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b	Unhealthy	Readiness probe failed: Get "http://10.132.0.55:8080/v2/models/isvc-xgboost-v2-runtime/ready": dial tcp 10.132.0.55:8080: connect: connection refused
19:25:48	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b	Unhealthy	Readiness probe failed: Get "https://10.132.0.55:8643/healthz": dial tcp 10.132.0.55:8643: connect: connection refused
19:25:49	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-predictor-fcc668bc from 0 to 1
19:25:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
19:25:50 (x2)	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-8t6kd	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-v2-predictor-serving-cert" not found
19:25:50	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-predictor-fcc668bc	SuccessfulCreate	Created pod: isvc-xgboost-v2-predictor-fcc668bc-8t6kd
19:25:52	kserve-ci-e2e-test	multus	isvc-xgboost-v2-predictor-fcc668bc-8t6kd	AddedInterface	Add eth0 [10.134.0.52/23] from ovn-kubernetes
19:25:52	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-8t6kd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:25:52	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-8t6kd	Created	Created container: storage-initializer
19:25:52	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-8t6kd	Started	Started container storage-initializer
19:25:53 (x9)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-888mm	Unhealthy	Readiness probe failed: dial tcp 10.132.0.54:8080: connect: connection refused
19:25:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-8t6kd	Pulled	Container image "kserve/xgbserver:latest" already present on machine
19:25:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-8t6kd	Created	Created container: kserve-container
19:25:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-8t6kd	Started	Started container kserve-container
19:25:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-8t6kd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:25:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-8t6kd	Started	Started container kube-rbac-proxy
19:25:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-8t6kd	Created	Created container: kube-rbac-proxy
19:26:03 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:26:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-runtime] is Ready
19:26:05 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:26:05 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:26:06	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-888mm	Killing	Stopping container kube-rbac-proxy
19:26:06	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-888mm	Killing	Stopping container kserve-container
19:26:08	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-888mm	Unhealthy	Readiness probe failed: Get "https://10.132.0.54:8643/healthz": dial tcp 10.132.0.54:8643: connect: connection refused
19:26:11	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-runtime-predictor-7f4957c5d5 from 0 to 1
19:26:11	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5	SuccessfulCreate	Created pod: isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh
19:26:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
19:26:11	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
19:26:12	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh	Started	Started container storage-initializer
19:26:12	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh	Created	Created container: storage-initializer
19:26:12	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh	AddedInterface	Add eth0 [10.132.0.56/23] from ovn-kubernetes
19:26:12	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:26:17	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh	Started	Started container kube-rbac-proxy
19:26:17	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh	Started	Started container kserve-container
19:26:17	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh	Created	Created container: kserve-container
19:26:17	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh	Created	Created container: kube-rbac-proxy
19:26:17	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:26:17	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:26:35 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:26:35 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:26:41 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:26:41 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:26:54 (x8)	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-8t6kd	Unhealthy	Readiness probe failed: dial tcp 10.134.0.52:8080: connect: connection refused
19:26:54 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:26:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-runtime] is Ready
19:26:56	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:26:56	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:27:01	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh	Killing	Stopping container kube-rbac-proxy
19:27:01	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh	Killing	Stopping container kserve-container
19:27:04 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:27:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	InferenceServiceReady	InferenceService [isvc-xgboost-v2] is Ready
19:27:05 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:27:05 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:27:06	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
19:27:06	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5	SuccessfulCreate	Created pod: isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj
19:27:06	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5 from 0 to 1
19:27:07	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj	AddedInterface	Add eth0 [10.134.0.53/23] from ovn-kubernetes
19:27:07	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj	Created	Created container: storage-initializer
19:27:07	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:27:07	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj	Started	Started container storage-initializer
19:27:10	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-8t6kd	Killing	Stopping container kserve-container
19:27:10	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-8t6kd	Killing	Stopping container kube-rbac-proxy
19:27:11	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj	Pulled	Container image "kserve/lgbserver:latest" already present on machine
19:27:11	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj	Created	Created container: kube-rbac-proxy
19:27:11	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj	Created	Created container: kserve-container
19:27:11	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj	Started	Started container kserve-container
19:27:11	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:27:11	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj	Started	Started container kube-rbac-proxy
19:27:14 (x2)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj	Unhealthy	Readiness probe failed: dial tcp 10.134.0.53:8080: connect: connection refused
19:27:15	kserve-ci-e2e-test	multus	isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx	AddedInterface	Add eth0 [10.134.0.54/23] from ovn-kubernetes
19:27:15	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-predictor-6ffbb5b99c	SuccessfulCreate	Created pod: isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx
19:27:15 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3": the object has been modified; please apply your changes to the latest version and try again
19:27:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3": the object has been modified; please apply your changes to the latest version and try again
19:27:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx	Started	Started container storage-initializer
19:27:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:27:15	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-predictor-6ffbb5b99c from 0 to 1
19:27:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx	Created	Created container: storage-initializer
19:27:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx	Created	Created container: kserve-container
19:27:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:27:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx	Created	Created container: kube-rbac-proxy
19:27:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx	Started	Started container kube-rbac-proxy
19:27:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx	Started	Started container kserve-container
19:27:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" already present on machine
19:27:19 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:27:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-kserve] is Ready
19:27:21	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:27:21	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:27:26	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj	Killing	Stopping container kube-rbac-proxy
19:27:26	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ltwfj	Killing	Stopping container kserve-container
19:27:30	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:27:30	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:27:31	kserve-ci-e2e-test	deployment-controller	isvc-mlflow-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-mlflow-v2-runtime-predictor-6456f94494 from 0 to 1
19:27:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-mlflow-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-mlflow-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
19:27:31	kserve-ci-e2e-test	replicaset-controller	isvc-mlflow-v2-runtime-predictor-6456f94494	SuccessfulCreate	Created pod: isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s
19:27:32	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:27:32	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s	Created	Created container: storage-initializer
19:27:32	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s	Started	Started container storage-initializer
19:27:32	kserve-ci-e2e-test	multus	isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s	AddedInterface	Add eth0 [10.132.0.57/23] from ovn-kubernetes
19:27:37	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:27:37	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s	Created	Created container: kserve-container
19:27:37	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s	Created	Created container: kube-rbac-proxy
19:27:37	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s	Started	Started container kserve-container
19:27:37	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:27:37	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s	Started	Started container kube-rbac-proxy
19:28:01 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:28:01 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:28:15 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:28:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	InferenceServiceReady	InferenceService [isvc-mlflow-v2-runtime] is Ready
19:28:16	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:28:16	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:28:21	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s	Killing	Stopping container kserve-container
19:28:21	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s	Killing	Stopping container kube-rbac-proxy
19:28:26	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-mcp-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-mcp-predictor-677dfdb8d8 from 0 to 1
19:28:27	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-mcp-predictor-677dfdb8d8	SuccessfulCreate	Created pod: isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc
19:28:27 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-mcp": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-mcp": the object has been modified; please apply your changes to the latest version and try again
19:28:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
19:28:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc	Started	Started container storage-initializer
19:28:28	kserve-ci-e2e-test	multus	isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc	AddedInterface	Add eth0 [10.132.0.58/23] from ovn-kubernetes
19:28:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:28:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc	Created	Created container: storage-initializer
19:28:30 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:28:30 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:28:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc	Started	Started container kserve-container
19:28:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" already present on machine
19:28:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc	Pulling	Pulling image "quay.io/opendatahub/kserve-agent:latest"
19:28:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc	Created	Created container: kserve-container
19:28:33	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	InferenceServiceReady	InferenceService [isvc-sklearn-s3] is Ready
19:28:33 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:28:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:28:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent:latest" in 2.826s (2.826s including waiting). Image size: 239481897 bytes.
19:28:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc	Created	Created container: kserve-agent
19:28:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc	Started	Started container kserve-agent
19:28:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc	Started	Started container kube-rbac-proxy
19:28:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc	Created	Created container: kube-rbac-proxy
19:28:39	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44
19:28:39	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf from 0 to 1
19:28:39 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-pass": the object has been modified; please apply your changes to the latest version and try again
19:28:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-pass": the object has been modified; please apply your changes to the latest version and try again
19:28:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx	Killing	Stopping container kube-rbac-proxy
19:28:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
19:28:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx	Killing	Stopping container kserve-container
19:28:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:28:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx	Unhealthy	Readiness probe failed: Get "https://10.134.0.54:8643/healthz": dial tcp 10.134.0.54:8643: connect: connection refused
19:28:40	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44	AddedInterface	Add eth0 [10.134.0.55/23] from ovn-kubernetes
19:28:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44	Created	Created container: storage-initializer
19:28:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44	Started	Started container storage-initializer
19:28:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" already present on machine
19:28:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:28:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44	Created	Created container: kube-rbac-proxy
19:28:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44	Started	Started container kserve-container
19:28:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44	Created	Created container: kserve-container
19:28:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44	Started	Started container kube-rbac-proxy
19:28:43 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6ffbb5b99c-f6jrx	Unhealthy	Readiness probe failed: dial tcp 10.134.0.54:8080: connect: connection refused
19:28:54	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:28:54	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:29:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc	Unhealthy	Liveness probe failed: Get "http://10.132.0.58:8080/v1/models/isvc-sklearn-mcp": dial tcp 10.132.0.58:8080: connect: connection refused
19:29:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	InferenceServiceReady	InferenceService [isvc-sklearn-mcp] is Ready
19:29:41 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:29:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc	Killing	Stopping container kserve-agent
19:29:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc	Killing	Stopping container kserve-container
19:29:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc	Killing	Stopping container kube-rbac-proxy
19:29:52	kserve-ci-e2e-test	multus	isvc-paddle-predictor-6d5c9fd4f6-q7648	AddedInterface	Add eth0 [10.134.0.56/23] from ovn-kubernetes
19:29:52	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-q7648	Created	Created container: storage-initializer
19:29:52	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-predictor-6d5c9fd4f6	SuccessfulCreate	Created pod: isvc-paddle-predictor-6d5c9fd4f6-q7648
19:29:52	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	UpdateFailed	Failed to update status for InferenceService "isvc-paddle": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle": the object has been modified; please apply your changes to the latest version and try again
19:29:52	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-q7648	Started	Started container storage-initializer
19:29:52	kserve-ci-e2e-test	deployment-controller	isvc-paddle-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-predictor-6d5c9fd4f6 from 0 to 1
19:29:52	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-q7648	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:29:54 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:29:54 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:29:55 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44	Unhealthy	Readiness probe failed: dial tcp 10.134.0.55:8080: connect: connection refused
19:29:58	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-q7648	Pulling	Pulling image "kserve/paddleserver:latest"
19:30:05 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:30:05	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-global-pass] is Ready
19:30:09	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-q7648	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:30:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44	Killing	Stopping container kube-rbac-proxy
19:30:09	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-q7648	Pulled	Successfully pulled image "kserve/paddleserver:latest" in 11.469s (11.469s including waiting). Image size: 1175702273 bytes.
19:30:09	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-q7648	Created	Created container: kserve-container
19:30:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44	Unhealthy	Readiness probe failed: Get "https://10.134.0.55:8643/healthz": dial tcp 10.134.0.55:8643: connect: connection refused
19:30:09	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-q7648	Started	Started container kserve-container
19:30:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6c88656fbf-wgz44	Killing	Stopping container kserve-container
19:30:09	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-q7648	Created	Created container: kube-rbac-proxy
19:30:09	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-q7648	Started	Started container kube-rbac-proxy
19:30:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-fail": the object has been modified; please apply your changes to the latest version and try again
19:30:10	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7
19:30:10	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549 from 0 to 1
19:30:11 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc	Unhealthy	Readiness probe failed: Get "http://10.132.0.58:8080/v1/models/isvc-sklearn-mcp": dial tcp 10.132.0.58:8080: connect: connection refused
19:30:11	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7	AddedInterface	Add eth0 [10.132.0.59/23] from ovn-kubernetes
19:30:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
19:30:16 (x6)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc	Unhealthy	Readiness probe failed: Get "https://10.132.0.58:8643/healthz": dial tcp 10.132.0.58:8643: connect: connection refused
19:30:17 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7	Started	Started container storage-initializer
19:30:17 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:30:17 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7	Created	Created container: storage-initializer
19:30:17 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:30:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7	Killing	Stopping container storage-initializer
19:30:21	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-pass": the object has been modified; please apply your changes to the latest version and try again
19:30:21 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-pass": the object has been modified; please apply your changes to the latest version and try again
19:30:21	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc from 0 to 1
19:30:21	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt
19:30:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:30:22	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt	AddedInterface	Add eth0 [10.134.0.57/23] from ovn-kubernetes
19:30:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt	Started	Started container storage-initializer
19:30:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt	Created	Created container: storage-initializer
19:30:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt	Started	Started container kserve-container
19:30:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt	Created	Created container: kserve-container
19:30:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" already present on machine
19:30:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:30:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt	Created	Created container: kube-rbac-proxy
19:30:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt	Started	Started container kube-rbac-proxy
19:30:47 (x6)	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-q7648	Unhealthy	Readiness probe failed: dial tcp 10.134.0.56:8080: connect: connection refused
19:30:52 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:30:52 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:30:57	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	InferenceServiceReady	InferenceService [isvc-paddle] is Ready
19:30:57 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:31:03	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-q7648	Killing	Stopping container kube-rbac-proxy
19:31:03	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-q7648	Killing	Stopping container kserve-container
19:31:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:31:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:31:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-runtime": the object has been modified; please apply your changes to the latest version and try again
19:31:08	kserve-ci-e2e-test	deployment-controller	isvc-paddle-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-runtime-predictor-58b99fff76 from 0 to 1
19:31:08	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-runtime-predictor-58b99fff76	SuccessfulCreate	Created pod: isvc-paddle-runtime-predictor-58b99fff76-g5mt6
19:31:09	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-g5mt6	Created	Created container: storage-initializer
19:31:09	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-g5mt6	Started	Started container storage-initializer
19:31:09	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-g5mt6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:31:09	kserve-ci-e2e-test	multus	isvc-paddle-runtime-predictor-58b99fff76-g5mt6	AddedInterface	Add eth0 [10.134.0.58/23] from ovn-kubernetes
19:31:13	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-g5mt6	Started	Started container kube-rbac-proxy
19:31:13	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-g5mt6	Created	Created container: kube-rbac-proxy
19:31:13	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-g5mt6	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:31:13	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-g5mt6	Started	Started container kserve-container
19:31:13	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-g5mt6	Created	Created container: kserve-container
19:31:13	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-g5mt6	Pulled	Container image "kserve/paddleserver:latest" already present on machine
19:31:31 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt	Unhealthy	Readiness probe failed: dial tcp 10.134.0.57:8080: connect: connection refused
19:31:36 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:31:36 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:31:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt	Killing	Stopping container kserve-container
19:31:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-custom-pass] is Ready
19:31:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt	Killing	Stopping container kube-rbac-proxy
19:31:41 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:31:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-fail": the object has been modified; please apply your changes to the latest version and try again
19:31:42	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85 from 0 to 1
19:31:43	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5
19:31:43	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5	AddedInterface	Add eth0 [10.132.0.60/23] from ovn-kubernetes
19:31:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-56dcbbb7cc-584qt	Unhealthy	Readiness probe failed: Get "https://10.134.0.57:8643/healthz": dial tcp 10.134.0.57:8643: connect: connection refused
19:31:50 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:31:50 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5	Started	Started container storage-initializer
19:31:50 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5	Created	Created container: storage-initializer
19:31:50 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:31:53 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:31:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5	Killing	Stopping container storage-initializer
19:31:53 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:31:54	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4 from 0 to 1
19:31:54 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-pass": the object has been modified; please apply your changes to the latest version and try again
19:31:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-pass": the object has been modified; please apply your changes to the latest version and try again
19:31:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w	Started	Started container storage-initializer
19:31:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:31:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w	Created	Created container: storage-initializer
19:31:54	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w	AddedInterface	Add eth0 [10.134.0.59/23] from ovn-kubernetes
19:31:54	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w
19:31:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w	Created	Created container: kube-rbac-proxy
19:31:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" already present on machine
19:31:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w	Started	Started container kube-rbac-proxy
19:31:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:31:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w	Started	Started container kserve-container
19:31:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w	Created	Created container: kserve-container
19:32:01 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:32:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	InferenceServiceReady	InferenceService [isvc-paddle-runtime] is Ready
19:32:08	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:32:08	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:32:09	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-g5mt6	Killing	Stopping container kserve-container
19:32:09	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-g5mt6	Killing	Stopping container kube-rbac-proxy
19:32:11	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-g5mt6	Unhealthy	Readiness probe failed: Get "https://10.134.0.58:8643/healthz": dial tcp 10.134.0.58:8643: connect: connection refused
19:32:11 (x7)	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-g5mt6	Unhealthy	Readiness probe failed: dial tcp 10.134.0.58:8080: connect: connection refused
19:32:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
19:32:14	kserve-ci-e2e-test	deployment-controller	isvc-paddle-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-v2-kserve-predictor-64d4c74b4d from 0 to 1
19:32:14	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-v2-kserve-predictor-64d4c74b4d	SuccessfulCreate	Created pod: isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv
19:32:15	kserve-ci-e2e-test	multus	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv	AddedInterface	Add eth0 [10.134.0.60/23] from ovn-kubernetes
19:32:15	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:32:15	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv	Created	Created container: storage-initializer
19:32:15	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv	Started	Started container storage-initializer
19:32:19	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv	Pulled	Container image "kserve/paddleserver:latest" already present on machine
19:32:19	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv	Created	Created container: kserve-container
19:32:19	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv	Started	Started container kserve-container
19:32:19	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:32:19	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv	Created	Created container: kube-rbac-proxy
19:32:19	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv	Started	Started container kube-rbac-proxy
19:32:24 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:32:24 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:32:59 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:32:59 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:33:07	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	InferenceServiceReady	InferenceService [isvc-paddle-v2-kserve] is Ready
19:33:08 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:33:09 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:33:09 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:33:14	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:33:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-serving-pass] is Ready
19:33:14 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:33:14	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:33:16	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv	Killing	Stopping container kube-rbac-proxy
19:33:16	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv	Killing	Stopping container kserve-container
19:33:17 (x7)	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv	Unhealthy	Readiness probe failed: dial tcp 10.134.0.60:8080: connect: connection refused
19:33:17	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jgvgv	Unhealthy	Readiness probe failed: Get "https://10.134.0.60:8643/healthz": dial tcp 10.134.0.60:8643: connect: connection refused
19:33:21	kserve-ci-e2e-test	deployment-controller	isvc-pmml-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-predictor-8674795f59 from 0 to 1
19:33:21	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml": the object has been modified; please apply your changes to the latest version and try again
19:33:21	kserve-ci-e2e-test	multus	isvc-pmml-predictor-8674795f59-hcrpq	AddedInterface	Add eth0 [10.134.0.61/23] from ovn-kubernetes
19:33:21 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	UpdateFailed	Failed to update status for InferenceService "isvc-pmml": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml": the object has been modified; please apply your changes to the latest version and try again
19:33:21	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-predictor-8674795f59	SuccessfulCreate	Created pod: isvc-pmml-predictor-8674795f59-hcrpq
19:33:22	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-hcrpq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:33:22	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-hcrpq	Started	Started container storage-initializer
19:33:22	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-hcrpq	Created	Created container: storage-initializer
19:33:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w	Killing	Stopping container kube-rbac-proxy
19:33:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w	Killing	Stopping container kserve-container
19:33:24 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w	Unhealthy	Readiness probe failed: dial tcp 10.134.0.59:8080: connect: connection refused
19:33:25	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-fail": the object has been modified; please apply your changes to the latest version and try again
19:33:25	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj	AddedInterface	Add eth0 [10.134.0.62/23] from ovn-kubernetes
19:33:25	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj
19:33:25	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76 from 0 to 1
19:33:25 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-fail": the object has been modified; please apply your changes to the latest version and try again
19:33:26	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-hcrpq	Pulling	Pulling image "kserve/pmmlserver:latest"
19:33:29 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-cd986c6c4-9tq5w	Unhealthy	Readiness probe failed: Get "https://10.134.0.59:8643/healthz": dial tcp 10.134.0.59:8643: connect: connection refused
19:33:33	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-hcrpq	Pulled	Successfully pulled image "kserve/pmmlserver:latest" in 7.148s (7.148s including waiting). Image size: 826884477 bytes.
19:33:33	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-hcrpq	Created	Created container: kube-rbac-proxy
19:33:33 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:33:33	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-hcrpq	Created	Created container: kserve-container
19:33:33 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj	Started	Started container storage-initializer
19:33:33 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj	Created	Created container: storage-initializer
19:33:33	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-hcrpq	Started	Started container kserve-container
19:33:33	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-hcrpq	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:33:33	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-hcrpq	Started	Started container kube-rbac-proxy
19:33:33 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:33:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-554ddb7c76-7n7bj	Killing	Stopping container storage-initializer
19:33:36	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-runtime-predictor-885fc8777	SuccessfulCreate	Created pod: isvc-pmml-runtime-predictor-885fc8777-qz2qm
19:33:36	kserve-ci-e2e-test	deployment-controller	isvc-pmml-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-runtime-predictor-885fc8777 from 0 to 1
19:33:36 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-runtime": the object has been modified; please apply your changes to the latest version and try again
19:33:36	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-runtime": the object has been modified; please apply your changes to the latest version and try again
19:33:36	kserve-ci-e2e-test	multus	isvc-pmml-runtime-predictor-885fc8777-qz2qm	AddedInterface	Add eth0 [10.134.0.63/23] from ovn-kubernetes
19:33:37	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-qz2qm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:33:37	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-qz2qm	Created	Created container: storage-initializer
19:33:37	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-qz2qm	Started	Started container storage-initializer
19:33:41	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-qz2qm	Started	Started container kserve-container
19:33:41	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-qz2qm	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
19:33:41	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-qz2qm	Created	Created container: kserve-container
19:33:41	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-qz2qm	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:33:41	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-qz2qm	Created	Created container: kube-rbac-proxy
19:33:41	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-qz2qm	Started	Started container kube-rbac-proxy
19:34:21 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:34:21 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:34:36 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:34:36 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:34:51 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:34:51 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-hcrpq	Unhealthy	Readiness probe failed: dial tcp 10.134.0.61:8080: connect: connection refused
19:34:51 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:34:56 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:34:56	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	InferenceServiceReady	InferenceService [isvc-pmml] is Ready
19:35:02	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-hcrpq	Killing	Stopping container kserve-container
19:35:02	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-hcrpq	Killing	Stopping container kube-rbac-proxy
19:35:06 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:35:06 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:35:07 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-404a29	UpdateFailed	Failed to update status for InferenceService "isvc-primary-404a29": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-404a29": the object has been modified; please apply your changes to the latest version and try again
19:35:07 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-404a29	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-404a29": the object has been modified; please apply your changes to the latest version and try again
19:35:07	kserve-ci-e2e-test	replicaset-controller	isvc-primary-404a29-predictor-8574c4b8bf	SuccessfulCreate	Created pod: isvc-primary-404a29-predictor-8574c4b8bf-kvtlt
19:35:07	kserve-ci-e2e-test	deployment-controller	isvc-primary-404a29-predictor	ScalingReplicaSet	Scaled up replica set isvc-primary-404a29-predictor-8574c4b8bf from 0 to 1
19:35:08	kserve-ci-e2e-test	kubelet	isvc-primary-404a29-predictor-8574c4b8bf-kvtlt	Started	Started container storage-initializer
19:35:08	kserve-ci-e2e-test	kubelet	isvc-primary-404a29-predictor-8574c4b8bf-kvtlt	Created	Created container: storage-initializer
19:35:08	kserve-ci-e2e-test	kubelet	isvc-primary-404a29-predictor-8574c4b8bf-kvtlt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:35:08	kserve-ci-e2e-test	multus	isvc-primary-404a29-predictor-8574c4b8bf-kvtlt	AddedInterface	Add eth0 [10.134.0.64/23] from ovn-kubernetes
19:35:09	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	InferenceServiceReady	InferenceService [isvc-pmml-runtime] is Ready
19:35:09 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:35:12	kserve-ci-e2e-test	kubelet	isvc-primary-404a29-predictor-8574c4b8bf-kvtlt	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:35:12	kserve-ci-e2e-test	kubelet	isvc-primary-404a29-predictor-8574c4b8bf-kvtlt	Started	Started container kserve-container
19:35:12	kserve-ci-e2e-test	kubelet	isvc-primary-404a29-predictor-8574c4b8bf-kvtlt	Created	Created container: kserve-container
19:35:12	kserve-ci-e2e-test	kubelet	isvc-primary-404a29-predictor-8574c4b8bf-kvtlt	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1986" already present on machine
19:35:12	kserve-ci-e2e-test	kubelet	isvc-primary-404a29-predictor-8574c4b8bf-kvtlt	Created	Created container: kube-rbac-proxy
19:35:12	kserve-ci-e2e-test	kubelet	isvc-primary-404a29-predictor-8574c4b8bf-kvtlt	Started	Started container kube-rbac-proxy
19:35:17	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-qz2qm	Killing	Stopping container kserve-container
19:35:17	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-qz2qm	Killing	Stopping container kube-rbac-proxy
19:35:19	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-qz2qm	Unhealthy	Readiness probe failed: Get "https://10.134.0.63:8643/healthz": dial tcp 10.134.0.63:8643: connect: connection refused
19:35:19 (x11)	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-qz2qm	Unhealthy	Readiness probe failed: dial tcp 10.134.0.63:8080: connect: connection refused
19:35:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
19:35:22	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-v2-kserve-predictor-84cb55486c	SuccessfulCreate	Created pod: isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx
19:35:22	kserve-ci-e2e-test	deployment-controller	isvc-pmml-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-v2-kserve-predictor-84cb55486c from 0 to 1
19:35:23	kserve-ci-e2e-test	multus	isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx	AddedInterface	Add eth0 [10.134.0.65/23] from ovn-kubernetes
19:35:23	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx	Created	Created container: storage-initializer
19:35:23	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:35:23	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx	Started	Started container storage-initializer
19:35:27	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
19:35:27	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx	Created	Created container: kserve-container
19:35:27	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx	Started	Started container kserve-container
19:35:27	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:35:27	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx	Created	Created container: kube-rbac-proxy
19:35:27	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx	Started	Started container kube-rbac-proxy
19:35:52 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-404a29-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:35:52 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-404a29-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:36:07 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:36:07 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:36:19 (x9)	kserve-ci-e2e-test	kubelet	isvc-primary-404a29-predictor-8574c4b8bf-kvtlt	Unhealthy	Readiness probe failed: dial tcp 10.134.0.64:8080: connect: connection refused
19:36:27 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-404a29	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:36:27	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-404a29	InferenceServiceReady	InferenceService [isvc-primary-404a29] is Ready
19:36:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-404a29	UpdateFailed	Failed to update status for InferenceService "isvc-secondary-404a29": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-404a29": the object has been modified; please apply your changes to the latest version and try again
19:36:42	kserve-ci-e2e-test	deployment-controller	isvc-secondary-404a29-predictor	ScalingReplicaSet	Scaled up replica set isvc-secondary-404a29-predictor-544c4b6766 from 0 to 1
19:36:42	kserve-ci-e2e-test	replicaset-controller	isvc-secondary-404a29-predictor-544c4b6766	SuccessfulCreate	Created pod: isvc-secondary-404a29-predictor-544c4b6766-nlpb8
19:36:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-404a29	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-404a29": the object has been modified; please apply your changes to the latest version and try again
19:36:43	kserve-ci-e2e-test	kubelet	isvc-secondary-404a29-predictor-544c4b6766-nlpb8	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-secondary-404a29-predictor-serving-cert" not found
19:36:44 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx	Unhealthy	Readiness probe failed: dial tcp 10.134.0.65:8080: connect: connection refused
19:36:44	kserve-ci-e2e-test	multus	isvc-secondary-404a29-predictor-544c4b6766-nlpb8	AddedInterface	Add eth0 [10.134.0.66/23] from ovn-kubernetes
19:36:47 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-404a29-predictor-544c4b6766-nlpb8	Created	Created container: storage-initializer
19:36:47 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-404a29-predictor-544c4b6766-nlpb8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5e3f2c9028a91a0c41fb60e357bf2ecf428704c551e5155589c954076f13371d" already present on machine
19:36:47 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-404a29-predictor-544c4b6766-nlpb8	Started	Started container storage-initializer
19:36:50 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-404a29	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:36:50 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:36:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	InferenceServiceReady	InferenceService [isvc-pmml-v2-kserve] is Ready
19:36:50	kserve-ci-e2e-test	kubelet	isvc-secondary-404a29-predictor-544c4b6766-nlpb8	BackOff	Back-off restarting failed container storage-initializer in pod isvc-secondary-404a29-predictor-544c4b6766-nlpb8_kserve-ci-e2e-test(480acdec-db55-405b-88c6-ab5bab2a3c6f)
19:36:52 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-404a29-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:36:52 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-404a29-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:36:52 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:36:52 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:36:53	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx	Killing	Stopping container kserve-container
19:36:53	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx	Killing	Stopping container kube-rbac-proxy
19:36:54	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-qh8mx	Unhealthy	Readiness probe failed: Get "https://10.134.0.65:8643/healthz": dial tcp 10.134.0.65:8643: connect: connection refused
19:36:57	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-404a29-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:36:57	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-404a29-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:36:59	kserve-ci-e2e-test	kubelet	isvc-primary-404a29-predictor-8574c4b8bf-kvtlt	Killing	Stopping container kserve-container
19:36:59	kserve-ci-e2e-test	kubelet	isvc-primary-404a29-predictor-8574c4b8bf-kvtlt	Killing	Stopping container kube-rbac-proxy
19:36:59	kserve-ci-e2e-test	kubelet	isvc-primary-404a29-predictor-8574c4b8bf-kvtlt	Unhealthy	Readiness probe failed: Get "https://10.134.0.64:8643/healthz": dial tcp 10.134.0.64:8643: connect: connection refused