Events

Time	Namespace	Component	RelatedObject	Reason	Message
16:47:03	kserve-ci-e2e-test		isvc-canary-create-v2-predictor-96f77bdbd-v25rb	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-v25rb to ip-10-0-143-197.ec2.internal
16:45:35	kserve-ci-e2e-test		isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w to ip-10-0-143-197.ec2.internal
16:48:58	kserve-ci-e2e-test		isvc-tensorflow-predictor-547dfc5669-lf8vp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-lf8vp to ip-10-0-143-197.ec2.internal
16:44:42	kserve-ci-e2e-test		isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9 to ip-10-0-142-248.ec2.internal
17:07:09	kserve-ci-e2e-test		isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9 to ip-10-0-143-197.ec2.internal
17:04:47	kserve-ci-e2e-test		isvc-paddle-predictor-6d5c9fd4f6-xf4dn	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xf4dn to ip-10-0-143-197.ec2.internal
16:45:47	kserve-ci-e2e-test		isvc-sklearn-v2-predictor-ffc44684-v5xxc	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-v5xxc to ip-10-0-143-197.ec2.internal
16:49:44	kserve-ci-e2e-test		isvc-tensorflow-runtime-predictor-855cfcd677-6fp45	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45 to ip-10-0-142-248.ec2.internal
17:08:16	kserve-ci-e2e-test		isvc-pmml-predictor-8674795f59-bnjbw	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw to ip-10-0-142-248.ec2.internal
16:47:22	kserve-ci-e2e-test		isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls to ip-10-0-143-197.ec2.internal
17:06:59	kserve-ci-e2e-test		isvc-pmml-runtime-predictor-885fc8777-lfvrb	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lfvrb to ip-10-0-143-197.ec2.internal
17:05:17	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8 to ip-10-0-143-197.ec2.internal
17:06:48	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl to ip-10-0-142-248.ec2.internal
17:02:02	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp to ip-10-0-142-248.ec2.internal
17:02:26	kserve-ci-e2e-test		isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8 to ip-10-0-142-248.ec2.internal
16:36:36	kserve-ci-e2e-test		isvc-autogluon-v2-predictor-9684b4cd9-ng6fr	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ng6fr to ip-10-0-143-197.ec2.internal
16:54:25	kserve-ci-e2e-test		isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q to ip-10-0-142-248.ec2.internal
16:59:09	kserve-ci-e2e-test		isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd to ip-10-0-143-197.ec2.internal
16:33:43	kserve-ci-e2e-test		isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl to ip-10-0-143-197.ec2.internal
16:51:59	kserve-ci-e2e-test		isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk to ip-10-0-142-248.ec2.internal
16:29:53	kserve-ci-e2e-test		isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r to ip-10-0-143-197.ec2.internal
17:08:55	kserve-ci-e2e-test		isvc-pmml-v2-kserve-predictor-84cb55486c-286wj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-286wj to ip-10-0-143-197.ec2.internal
17:12:27	kserve-ci-e2e-test		isvc-secondary-932c75-predictor-76c5bfcd-xvcc9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-secondary-932c75-predictor-76c5bfcd-xvcc9 to ip-10-0-143-197.ec2.internal
16:55:35	kserve-ci-e2e-test		isvc-canary-stop-predictor-5cdcf6545-kqh6q	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-kqh6q to ip-10-0-143-197.ec2.internal
16:53:59	kserve-ci-e2e-test		isvc-canary-rollback-predictor-6b8c75c7b9-s77ll	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-s77ll to ip-10-0-143-197.ec2.internal
16:48:49	kserve-ci-e2e-test		isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h to ip-10-0-143-197.ec2.internal
16:50:24	kserve-ci-e2e-test		isvc-canary-promote-v2-predictor-5b4b466b66-8v72b	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b to ip-10-0-142-248.ec2.internal
17:01:51	kserve-ci-e2e-test		isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68 to ip-10-0-143-197.ec2.internal
16:44:04	kserve-ci-e2e-test		isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2 to ip-10-0-143-197.ec2.internal
16:30:13	kserve-ci-e2e-test		isvc-predictive-xgboost-predictor-695f978978-rv59q	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q to ip-10-0-142-248.ec2.internal
16:59:07	kserve-ci-e2e-test		isvc-xgboost-v2-predictor-fcc668bc-nmnxm	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-nmnxm to ip-10-0-143-197.ec2.internal
16:51:59	kserve-ci-e2e-test		isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk to ip-10-0-143-197.ec2.internal
16:48:49	kserve-ci-e2e-test		isvc-canary-promote-predictor-7cf78f884f-s5xm7	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-s5xm7 to ip-10-0-143-197.ec2.internal
16:27:58	kserve-ci-e2e-test		isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r to ip-10-0-143-197.ec2.internal
16:58:01	kserve-ci-e2e-test		isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm to ip-10-0-142-248.ec2.internal
16:41:11	kserve-ci-e2e-test		isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2 to ip-10-0-143-197.ec2.internal
16:31:29	kserve-ci-e2e-test		message-dumper-predictor-6575bcfdfd-kr9lt	Scheduled	Successfully assigned kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kr9lt to ip-10-0-143-197.ec2.internal
17:03:21	kserve-ci-e2e-test		isvc-sklearn-mcp-predictor-8584bb4bfb-nh268	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-nh268 to ip-10-0-143-197.ec2.internal
16:39:04	kserve-ci-e2e-test		isvc-sklearn-predictor-9d87cbc64-2k995	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-2k995 to ip-10-0-143-197.ec2.internal
16:31:48	kserve-ci-e2e-test		isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw to ip-10-0-143-197.ec2.internal
16:48:49	kserve-ci-e2e-test		isvc-canary-promote-predictor-7cf78f884f-l6qgw	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw to ip-10-0-142-248.ec2.internal
17:10:42	kserve-ci-e2e-test		isvc-primary-932c75-predictor-bd6c947bf-bhkvw	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-primary-932c75-predictor-bd6c947bf-bhkvw to ip-10-0-143-197.ec2.internal
16:51:59	kserve-ci-e2e-test		isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj to ip-10-0-143-197.ec2.internal
16:35:19	kserve-ci-e2e-test		isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz to ip-10-0-143-197.ec2.internal
16:31:49	kserve-ci-e2e-test		isvc-logger-predictor-6bdfd88f5f-rtc26	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-rtc26 to ip-10-0-143-197.ec2.internal
16:33:24	kserve-ci-e2e-test		isvc-autogluon-v1-predictor-574b5b7495-kvpvl	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl to ip-10-0-142-248.ec2.internal
17:06:03	kserve-ci-e2e-test		isvc-paddle-runtime-predictor-58b99fff76-njqm2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-njqm2 to ip-10-0-143-197.ec2.internal
16:56:16	kserve-ci-e2e-test		isvc-xgboost-runtime-predictor-54dd488c85-v47xd	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-v47xd to ip-10-0-143-197.ec2.internal
16:47:03	kserve-ci-e2e-test		isvc-canary-create-predictor-6474b875b9-l68gc	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-l68gc to ip-10-0-143-197.ec2.internal
16:47:03	kserve-ci-e2e-test		isvc-canary-create-predictor-6474b875b9-zrzwc	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc to ip-10-0-142-248.ec2.internal
17:03:23	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw to ip-10-0-143-197.ec2.internal
17:03:34	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f to ip-10-0-142-248.ec2.internal
17:05:06	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf to ip-10-0-142-248.ec2.internal
17:00:32	kserve-ci-e2e-test		isvc-sklearn-s3-predictor-84d46f45-8fxg6	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-8fxg6 to ip-10-0-143-197.ec2.internal
16:50:31	kserve-ci-e2e-test		isvc-triton-predictor-5d7db58676-k4mg9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-k4mg9 to ip-10-0-143-197.ec2.internal
16:55:21	kserve-ci-e2e-test		xgboost-v2-mlserver-predictor-59944bcf4c-kn69w	Scheduled	Successfully assigned kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w to ip-10-0-142-248.ec2.internal
16:55:35	kserve-ci-e2e-test		isvc-canary-stop-predictor-5cdcf6545-mth9z	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z to ip-10-0-142-248.ec2.internal
16:46:23	kserve-ci-e2e-test		isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5 to ip-10-0-143-197.ec2.internal
16:55:35	kserve-ci-e2e-test		isvc-canary-stop-v2-predictor-596664c988-bsz9b	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-bsz9b to ip-10-0-143-197.ec2.internal
17:00:56	kserve-ci-e2e-test		isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f to ip-10-0-142-248.ec2.internal
16:40:49	kserve-ci-e2e-test		sklearn-v2-mlserver-predictor-7f568cdc6b-44txf	Scheduled	Successfully assigned kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf to ip-10-0-142-248.ec2.internal
16:36:54	kserve-ci-e2e-test		isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2 to ip-10-0-142-248.ec2.internal
16:28:13	kserve-ci-e2e-test		isvc-predictive-sklearn-predictor-5f849d69df-mllcs	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs to ip-10-0-142-248.ec2.internal
16:57:20	kserve-ci-e2e-test		isvc-lightgbm-predictor-54f85ddf88-nz97x	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nz97x to ip-10-0-143-197.ec2.internal
16:27:58	kserve-ci-e2e-test		isvc-init-fail-5d5066-predictor	ClusterIPNotAllocated	Cluster IP [IPv4]: 172.31.198.215 is not allocated; repairing
16:52:50	kserve-ci-e2e-test		isvc-xgboost-predictor-7566d65bf7-cmnxd	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd to ip-10-0-142-248.ec2.internal
16:43:14	kserve-ci-e2e-test		isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k to ip-10-0-143-197.ec2.internal
16:27:59	kserve-ci-e2e-test		isvc-init-fail-5d5066-predictor-576dcc8968-gkw85	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85 to ip-10-0-142-248.ec2.internal
16:27:58	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-predictor-7485d8d7d4 from 0 to 1
16:27:58	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
16:27:58	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-predictor-7485d8d7d4	SuccessfulCreate	Created pod: isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r
16:27:58	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
16:27:59	kserve-ci-e2e-test	replicaset-controller	isvc-init-fail-5d5066-predictor-576dcc8968	SuccessfulCreate	Created pod: isvc-init-fail-5d5066-predictor-576dcc8968-gkw85
16:27:59	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r	AddedInterface	Add eth0 [10.133.0.21/23] from ovn-kubernetes
16:27:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948"
16:27:59	kserve-ci-e2e-test	deployment-controller	isvc-init-fail-5d5066-predictor	ScalingReplicaSet	Scaled up replica set isvc-init-fail-5d5066-predictor-576dcc8968 from 0 to 1
16:27:59	kserve-ci-e2e-test	multus	isvc-init-fail-5d5066-predictor-576dcc8968-gkw85	AddedInterface	Add eth0 [10.132.0.28/23] from ovn-kubernetes
16:28:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r	Started	Started container storage-initializer
16:28:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r	Created	Created container: storage-initializer
16:28:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" in 3.279s (3.279s including waiting). Image size: 306764719 bytes.
16:28:03 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-5d5066-predictor-576dcc8968-gkw85	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:28:03 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-5d5066-predictor-576dcc8968-gkw85	Started	Started container storage-initializer
16:28:03 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-5d5066-predictor-576dcc8968-gkw85	Created	Created container: storage-initializer
16:28:07 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-5d5066	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:28:07	kserve-ci-e2e-test	kubelet	isvc-init-fail-5d5066-predictor-576dcc8968-gkw85	BackOff	Back-off restarting failed container storage-initializer in pod isvc-init-fail-5d5066-predictor-576dcc8968-gkw85_kserve-ci-e2e-test(1a8de051-cb8a-4a41-8ef9-831a9ddc154e)
16:28:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928"
16:28:13	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-predictor-5f849d69df from 0 to 1
16:28:13	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-predictor-5f849d69df	SuccessfulCreate	Created pod: isvc-predictive-sklearn-predictor-5f849d69df-mllcs
16:28:14 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn": the object has been modified; please apply your changes to the latest version and try again
16:28:14	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-mllcs	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:28:14	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-mllcs	Started	Started container storage-initializer
16:28:14	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-mllcs	Created	Created container: storage-initializer
16:28:14	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-predictor-5f849d69df-mllcs	AddedInterface	Add eth0 [10.132.0.29/23] from ovn-kubernetes
16:28:18	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-mllcs	Pulling	Pulling image "kserve/predictiveserver:latest"
16:28:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" in 13.99s (13.99s including waiting). Image size: 1583654804 bytes.
16:28:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r	Started	Started container kserve-container
16:28:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
16:28:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r	Created	Created container: kserve-container
16:28:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r	Pulling	Pulling image "quay.io/opendatahub/kserve-agent@sha256:6bed5c97c6557bae64203d919b8edfd899fd57987abd7acf05784a97bd1466b4"
16:28:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r	Started	Started container kube-rbac-proxy
16:28:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r	Created	Created container: kube-rbac-proxy
16:28:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.377s (2.377s including waiting). Image size: 211946088 bytes.
16:28:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r	Created	Created container: agent
16:28:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:6bed5c97c6557bae64203d919b8edfd899fd57987abd7acf05784a97bd1466b4" in 2.499s (2.499s including waiting). Image size: 243402325 bytes.
16:28:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r	Started	Started container agent
16:28:42	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-mllcs	Started	Started container kserve-container
16:28:42	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-mllcs	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
16:28:42	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-mllcs	Created	Created container: kserve-container
16:28:42	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-mllcs	Pulled	Successfully pulled image "kserve/predictiveserver:latest" in 23.777s (23.777s including waiting). Image size: 2433581203 bytes.
16:28:44	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-mllcs	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.305s (2.305s including waiting). Image size: 211946088 bytes.
16:28:44	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-mllcs	Created	Created container: kube-rbac-proxy
16:28:44	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-mllcs	Started	Started container kube-rbac-proxy
16:28:58 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:28:58 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:29:13 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:29:13 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:29:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InferenceServiceReady	InferenceService [isvc-sklearn-batcher] is Ready
16:29:39 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:29:43 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:29:43 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:29:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r	Killing	Stopping container kube-rbac-proxy
16:29:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher-custom": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher-custom": the object has been modified; please apply your changes to the latest version and try again
16:29:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r	Killing	Stopping container agent
16:29:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r	Killing	Stopping container kserve-container
16:29:53	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-custom-predictor-8684fb99f9	SuccessfulCreate	Created pod: isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r
16:29:53	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-custom-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-custom-predictor-8684fb99f9 from 0 to 1
16:29:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher-custom": the object has been modified; please apply your changes to the latest version and try again
16:29:54	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r	AddedInterface	Add eth0 [10.133.0.22/23] from ovn-kubernetes
16:29:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:29:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r	Created	Created container: storage-initializer
16:29:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r	Started	Started container storage-initializer
16:29:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine
16:29:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r	Created	Created container: kserve-container
16:29:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r	Started	Started container kserve-container
16:29:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:29:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:6bed5c97c6557bae64203d919b8edfd899fd57987abd7acf05784a97bd1466b4" already present on machine
16:29:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r	Created	Created container: kube-rbac-proxy
16:29:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r	Started	Started container agent
16:29:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r	Created	Created container: agent
16:29:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r	Started	Started container kube-rbac-proxy
16:29:59 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:29:59 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:30:01 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:30:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	InferenceServiceReady	InferenceService [isvc-predictive-sklearn] is Ready
16:30:08	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-mllcs	Killing	Stopping container kube-rbac-proxy
16:30:08	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-mllcs	Killing	Stopping container kserve-container
16:30:08 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r	Unhealthy	Readiness probe failed: Get "https://10.133.0.21:8643/healthz": dial tcp 10.133.0.21:8643: connect: connection refused
16:30:09 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
16:30:09 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7485d8d7d4-hcw9r	Unhealthy	Readiness probe failed: dial tcp 10.133.0.21:8080: connect: connection refused
16:30:11 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-mllcs	Unhealthy	Readiness probe failed: dial tcp 10.132.0.29:8080: connect: connection refused
16:30:11	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-mllcs	Unhealthy	Readiness probe failed: Get "https://10.132.0.29:8643/healthz": dial tcp 10.132.0.29:8643: connect: connection refused
16:30:13	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-rv59q	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:30:13	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-rv59q	Started	Started container storage-initializer
16:30:13	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-predictor-695f978978-rv59q	AddedInterface	Add eth0 [10.132.0.30/23] from ovn-kubernetes
16:30:13	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-predictor-695f978978 from 0 to 1
16:30:13	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-predictor-695f978978	SuccessfulCreate	Created pod: isvc-predictive-xgboost-predictor-695f978978-rv59q
16:30:13	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost": the object has been modified; please apply your changes to the latest version and try again
16:30:13	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-rv59q	Created	Created container: storage-initializer
16:30:18	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-rv59q	Created	Created container: kserve-container
16:30:18	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-rv59q	Started	Started container kube-rbac-proxy
16:30:18	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-rv59q	Created	Created container: kube-rbac-proxy
16:30:18	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-rv59q	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:30:18	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-rv59q	Started	Started container kserve-container
16:30:18	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-rv59q	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
16:30:38 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:30:38 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:30:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:30:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:31:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	InferenceServiceReady	InferenceService [isvc-sklearn-batcher-custom] is Ready
16:31:16 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:31:23 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:31:23 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:31:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r	Killing	Stopping container kube-rbac-proxy
16:31:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r	Killing	Stopping container agent
16:31:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r	Killing	Stopping container kserve-container
16:31:29	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper": the object has been modified; please apply your changes to the latest version and try again
16:31:29	kserve-ci-e2e-test	multus	message-dumper-predictor-6575bcfdfd-kr9lt	AddedInterface	Add eth0 [10.133.0.23/23] from ovn-kubernetes
16:31:29	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-kr9lt	Pulling	Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display"
16:31:29 (x2)	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	UpdateFailed	Failed to update status for InferenceService "message-dumper": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper": the object has been modified; please apply your changes to the latest version and try again
16:31:29	kserve-ci-e2e-test	deployment-controller	message-dumper-predictor	ScalingReplicaSet	Scaled up replica set message-dumper-predictor-6575bcfdfd from 0 to 1
16:31:29	kserve-ci-e2e-test	replicaset-controller	message-dumper-predictor-6575bcfdfd	SuccessfulCreate	Created pod: message-dumper-predictor-6575bcfdfd-kr9lt
16:31:31	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-kr9lt	Started	Started container kube-rbac-proxy
16:31:31	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-kr9lt	Pulled	Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.4s (1.4s including waiting). Image size: 14813193 bytes.
16:31:31	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-kr9lt	Created	Created container: kserve-container
16:31:31	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-kr9lt	Started	Started container kserve-container
16:31:31	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-kr9lt	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:31:31	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-kr9lt	Created	Created container: kube-rbac-proxy
16:31:35 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:31:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	InferenceServiceReady	InferenceService [isvc-predictive-xgboost] is Ready
16:31:36 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
16:31:36 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r	Unhealthy	Readiness probe failed: dial tcp 10.133.0.22:5000: connect: connection refused
16:31:39 (x7)	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:31:39	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	InferenceServiceReady	InferenceService [message-dumper] is Ready
16:31:43	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-rv59q	Killing	Stopping container kserve-container
16:31:43	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-rv59q	Killing	Stopping container kube-rbac-proxy
16:31:43 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:31:43 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:31:44	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:31:44	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:31:45	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-rv59q	Unhealthy	Readiness probe failed: Get "https://10.132.0.30:8643/healthz": dial tcp 10.132.0.30:8643: connect: connection refused
16:31:45 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-rv59q	Unhealthy	Readiness probe failed: dial tcp 10.132.0.30:8080: connect: connection refused
16:31:46 (x4)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-8684fb99f9-tbq9r	Unhealthy	Readiness probe failed: Get "https://10.133.0.22:8643/healthz": dial tcp 10.133.0.22:8643: connect: connection refused
16:31:48	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-predictor-ff56bcbb5	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw
16:31:48	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-predictor-ff56bcbb5 from 0 to 1
16:31:48 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm": the object has been modified; please apply your changes to the latest version and try again
16:31:48	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm": the object has been modified; please apply your changes to the latest version and try again
16:31:49	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6bdfd88f5f-rtc26	Started	Started container storage-initializer
16:31:49	kserve-ci-e2e-test	replicaset-controller	isvc-logger-predictor-6bdfd88f5f	SuccessfulCreate	Created pod: isvc-logger-predictor-6bdfd88f5f-rtc26
16:31:49	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw	Created	Created container: storage-initializer
16:31:49	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:31:49	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6bdfd88f5f-rtc26	Created	Created container: storage-initializer
16:31:49	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw	AddedInterface	Add eth0 [10.133.0.24/23] from ovn-kubernetes
16:31:49	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6bdfd88f5f-rtc26	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:31:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	UpdateFailed	Failed to update status for InferenceService "isvc-logger": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger": the object has been modified; please apply your changes to the latest version and try again
16:31:49	kserve-ci-e2e-test	multus	isvc-logger-predictor-6bdfd88f5f-rtc26	AddedInterface	Add eth0 [10.133.0.25/23] from ovn-kubernetes
16:31:49	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw	Started	Started container storage-initializer
16:31:49	kserve-ci-e2e-test	deployment-controller	isvc-logger-predictor	ScalingReplicaSet	Scaled up replica set isvc-logger-predictor-6bdfd88f5f from 0 to 1
16:31:53	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw	Pulling	Pulling image "kserve/predictiveserver:latest"
16:31:54	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6bdfd88f5f-rtc26	Created	Created container: agent
16:31:54	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6bdfd88f5f-rtc26	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:31:54	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6bdfd88f5f-rtc26	Started	Started container kserve-container
16:31:54	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6bdfd88f5f-rtc26	Started	Started container agent
16:31:54	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6bdfd88f5f-rtc26	Created	Created container: kserve-container
16:31:54	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6bdfd88f5f-rtc26	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine
16:31:54	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6bdfd88f5f-rtc26	Created	Created container: kube-rbac-proxy
16:31:54	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6bdfd88f5f-rtc26	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:6bed5c97c6557bae64203d919b8edfd899fd57987abd7acf05784a97bd1466b4" already present on machine
16:31:54	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6bdfd88f5f-rtc26	Started	Started container kube-rbac-proxy
16:31:59	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:31:59	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:32:17	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw	Pulled	Successfully pulled image "kserve/predictiveserver:latest" in 23.078s (23.078s including waiting). Image size: 2433581203 bytes.
16:32:17	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw	Created	Created container: kube-rbac-proxy
16:32:17	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw	Started	Started container kserve-container
16:32:17	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw	Started	Started container kube-rbac-proxy
16:32:17	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw	Created	Created container: kserve-container
16:32:17	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:32:34 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:32:34 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:32:48 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:32:48 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:33:12	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	InferenceServiceReady	InferenceService [isvc-logger] is Ready
16:33:12 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:33:19 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:33:19 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:33:24 (x9)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw	Unhealthy	Readiness probe failed: dial tcp 10.133.0.24:8080: connect: connection refused
16:33:24	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6bdfd88f5f-rtc26	Killing	Stopping container kserve-container
16:33:24	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6bdfd88f5f-rtc26	Killing	Stopping container kube-rbac-proxy
16:33:24	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v1": the object has been modified; please apply your changes to the latest version and try again
16:33:24	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-kr9lt	Killing	Stopping container kserve-container
16:33:24	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6bdfd88f5f-rtc26	Killing	Stopping container agent
16:33:24	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v1-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v1-predictor-574b5b7495 from 0 to 1
16:33:24	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-kr9lt	Killing	Stopping container kube-rbac-proxy
16:33:24	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v1-predictor-574b5b7495	SuccessfulCreate	Created pod: isvc-autogluon-v1-predictor-574b5b7495-kvpvl
16:33:25	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-kvpvl	Created	Created container: storage-initializer
16:33:25	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-kvpvl	Started	Started container storage-initializer
16:33:25	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-kvpvl	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:33:25	kserve-ci-e2e-test	multus	isvc-autogluon-v1-predictor-574b5b7495-kvpvl	AddedInterface	Add eth0 [10.132.0.31/23] from ovn-kubernetes
16:33:32 (x11)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6bdfd88f5f-rtc26	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
16:33:32 (x11)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6bdfd88f5f-rtc26	Unhealthy	Readiness probe failed: dial tcp 10.133.0.25:8080: connect: connection refused
16:33:33 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:33:33 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:33:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm] is Ready
16:33:34 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:33:36 (x3)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6bdfd88f5f-rtc26	Unhealthy	Readiness probe failed: Get "https://10.133.0.25:8643/healthz": dial tcp 10.133.0.25:8643: connect: connection refused
16:33:38	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw	Killing	Stopping container kube-rbac-proxy
16:33:38	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw	Killing	Stopping container kserve-container
16:33:39	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw	Unhealthy	Readiness probe failed: Get "https://10.133.0.24:8643/healthz": dial tcp 10.133.0.24:8643: connect: connection refused
16:33:43	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
16:33:43	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-v2-predictor-745f9c8c9f	SuccessfulCreate	Created pod: isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl
16:33:43	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-v2-predictor-745f9c8c9f from 0 to 1
16:33:43 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
16:33:44	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
16:33:45	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl	AddedInterface	Add eth0 [10.133.0.26/23] from ovn-kubernetes
16:33:45	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:33:45	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl	Created	Created container: storage-initializer
16:33:45	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw	Unhealthy	Readiness probe failed: dial tcp 10.133.0.24:8080: i/o timeout
16:33:45	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-87ccw	Unhealthy	Readiness probe failed: Get "https://10.133.0.24:8643/healthz": context deadline exceeded
16:33:45	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl	Started	Started container storage-initializer
16:33:49	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:33:49	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl	Created	Created container: kserve-container
16:33:49	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl	Started	Started container kserve-container
16:33:49	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl	Created	Created container: kube-rbac-proxy
16:33:49	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
16:33:49	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl	Started	Started container kube-rbac-proxy
16:33:56	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-kvpvl	Pulling	Pulling image "kserve/autogluonserver:latest"
16:33:58	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:33:58	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:35:06	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	InferenceServiceReady	InferenceService [isvc-predictive-sklearn-v2] is Ready
16:35:06 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:35:13 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:35:13 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:35:13	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl	Killing	Stopping container kube-rbac-proxy
16:35:13	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl	Killing	Stopping container kserve-container
16:35:16 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl	Unhealthy	Readiness probe failed: Get "http://10.133.0.26:8080/v2/models/isvc-predictive-sklearn-v2/ready": dial tcp 10.133.0.26:8080: connect: connection refused
16:35:16	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-vkdhl	Unhealthy	Readiness probe failed: Get "https://10.133.0.26:8643/healthz": dial tcp 10.133.0.26:8643: connect: connection refused
16:35:19	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz	Started	Started container storage-initializer
16:35:19	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz	Created	Created container: storage-initializer
16:35:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
16:35:19	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:35:19	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-v2-predictor-868ff5dd6d from 0 to 1
16:35:19	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-v2-predictor-868ff5dd6d	SuccessfulCreate	Created pod: isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz
16:35:19	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz	AddedInterface	Add eth0 [10.133.0.27/23] from ovn-kubernetes
16:35:23	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:35:23	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
16:35:23	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz	Created	Created container: kserve-container
16:35:23	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz	Started	Started container kserve-container
16:35:24	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz	Created	Created container: kube-rbac-proxy
16:35:24	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz	Started	Started container kube-rbac-proxy
16:35:34	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:35:34	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:35:58	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-kvpvl	Pulled	Successfully pulled image "kserve/autogluonserver:latest" in 2m2.066s (2m2.066s including waiting). Image size: 8945753381 bytes.
16:35:58	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-kvpvl	Created	Created container: kserve-container
16:35:58	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-kvpvl	Started	Started container kserve-container
16:35:58	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-kvpvl	Started	Started container kube-rbac-proxy
16:35:58	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-kvpvl	Created	Created container: kube-rbac-proxy
16:35:58	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-kvpvl	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:36:05 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-kvpvl	Unhealthy	Readiness probe failed: dial tcp 10.132.0.31:8080: connect: connection refused
16:36:09 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:36:09 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:36:15 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:36:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	InferenceServiceReady	InferenceService [isvc-autogluon-v1] is Ready
16:36:24	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:36:24	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:36:30	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-kvpvl	Killing	Stopping container kserve-container
16:36:30	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-kvpvl	Killing	Stopping container kube-rbac-proxy
16:36:34 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:36:34 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:36:36	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2": the object has been modified; please apply your changes to the latest version and try again
16:36:36	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-predictor-9684b4cd9	SuccessfulCreate	Created pod: isvc-autogluon-v2-predictor-9684b4cd9-ng6fr
16:36:36	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-predictor-9684b4cd9 from 0 to 1
16:36:36	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-ng6fr	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-autogluon-v2-predictor-serving-cert" not found
16:36:37	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-ng6fr	Started	Started container storage-initializer
16:36:37	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-ng6fr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:36:37	kserve-ci-e2e-test	multus	isvc-autogluon-v2-predictor-9684b4cd9-ng6fr	AddedInterface	Add eth0 [10.133.0.28/23] from ovn-kubernetes
16:36:37	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-ng6fr	Created	Created container: storage-initializer
16:36:43	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	InferenceServiceReady	InferenceService [isvc-predictive-xgboost-v2] is Ready
16:36:43 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:36:49	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz	Killing	Stopping container kserve-container
16:36:49	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz	Killing	Stopping container kube-rbac-proxy
16:36:51	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz	Unhealthy	Readiness probe failed: Get "https://10.133.0.27:8643/healthz": dial tcp 10.133.0.27:8643: connect: connection refused
16:36:53 (x6)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-trwmz	Unhealthy	Readiness probe failed: Get "http://10.133.0.27:8080/v2/models/isvc-predictive-xgboost-v2/ready": dial tcp 10.133.0.27:8080: connect: connection refused
16:36:54	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-v2-predictor-545b7cc587 from 0 to 1
16:36:54	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-v2-predictor-545b7cc587	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2
16:36:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm-v2": the object has been modified; please apply your changes to the latest version and try again
16:36:54	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
16:36:55	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2	Started	Started container storage-initializer
16:36:55	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2	Created	Created container: storage-initializer
16:36:55	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:36:55	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2	AddedInterface	Add eth0 [10.132.0.32/23] from ovn-kubernetes
16:37:00	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2	Started	Started container kserve-container
16:37:00	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
16:37:00	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2	Started	Started container kube-rbac-proxy
16:37:00	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2	Created	Created container: kube-rbac-proxy
16:37:00	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:37:00	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2	Created	Created container: kserve-container
16:37:07	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-ng6fr	Pulling	Pulling image "kserve/autogluonserver:latest"
16:37:39 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:37:39 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:38:17 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:38:17	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm-v2] is Ready
16:38:24 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:38:24	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2	Killing	Stopping container kserve-container
16:38:24	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2	Killing	Stopping container kube-rbac-proxy
16:38:24 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:38:27 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2	Unhealthy	Readiness probe failed: Get "http://10.132.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready": dial tcp 10.132.0.32:8080: connect: connection refused
16:38:27	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2	Unhealthy	Readiness probe failed: Get "https://10.132.0.32:8643/healthz": dial tcp 10.132.0.32:8643: connect: connection refused
16:38:29	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-scheduler-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-scheduler-predictor-6bd688b666 from 0 to 1
16:38:29	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-scheduler": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
16:38:29	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
16:38:29	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-scheduler-predictor-6bd688b666	SuccessfulCreate	Created pod: isvc-sklearn-scheduler-predictor-6bd688b666-44zwb
16:38:30 (x6)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:39:04	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-predictor-9d87cbc64 from 0 to 1
16:39:04	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-predictor-9d87cbc64	SuccessfulCreate	Created pod: isvc-sklearn-predictor-9d87cbc64-2k995
16:39:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn": the object has been modified; please apply your changes to the latest version and try again
16:39:04 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn": the object has been modified; please apply your changes to the latest version and try again
16:39:18	kserve-ci-e2e-test	multus	isvc-sklearn-predictor-9d87cbc64-2k995	AddedInterface	Add eth0 [10.133.0.29/23] from ovn-kubernetes
16:39:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-9d87cbc64-2k995	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:39:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-ng6fr	Created	Created container: kube-rbac-proxy
16:39:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-9d87cbc64-2k995	Created	Created container: storage-initializer
16:39:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-ng6fr	Pulled	Successfully pulled image "kserve/autogluonserver:latest" in 2m12.076s (2m12.076s including waiting). Image size: 8945753381 bytes.
16:39:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-ng6fr	Created	Created container: kserve-container
16:39:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-ng6fr	Started	Started container kserve-container
16:39:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-ng6fr	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:39:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-ng6fr	Started	Started container kube-rbac-proxy
16:39:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-9d87cbc64-2k995	Started	Started container storage-initializer
16:39:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-9d87cbc64-2k995	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine
16:39:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-9d87cbc64-2k995	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:39:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-9d87cbc64-2k995	Created	Created container: kserve-container
16:39:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-9d87cbc64-2k995	Started	Started container kserve-container
16:39:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-9d87cbc64-2k995	Started	Started container kube-rbac-proxy
16:39:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-9d87cbc64-2k995	Created	Created container: kube-rbac-proxy
16:39:36 (x12)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:39:36 (x12)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:39:49 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:39:49 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:39:51	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:40:31 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-9d87cbc64-2k995	Unhealthy	Readiness probe failed: dial tcp 10.133.0.29:8080: connect: connection refused
16:40:34 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:40:34 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:40:42 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:40:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	InferenceServiceReady	InferenceService [isvc-sklearn] is Ready
16:40:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-9d87cbc64-2k995	Killing	Stopping container kube-rbac-proxy
16:40:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-9d87cbc64-2k995	Killing	Stopping container kserve-container
16:40:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-9d87cbc64-2k995	Unhealthy	Readiness probe failed: Get "https://10.133.0.29:8643/healthz": dial tcp 10.133.0.29:8643: connect: connection refused
16:40:49	kserve-ci-e2e-test	deployment-controller	sklearn-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set sklearn-v2-mlserver-predictor-7f568cdc6b from 0 to 1
16:40:49	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "sklearn-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "sklearn-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
16:40:49	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-44txf	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
16:40:49	kserve-ci-e2e-test	replicaset-controller	sklearn-v2-mlserver-predictor-7f568cdc6b	SuccessfulCreate	Created pod: sklearn-v2-mlserver-predictor-7f568cdc6b-44txf
16:40:50	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-44txf	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:40:50	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-44txf	Started	Started container storage-initializer
16:40:50	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-44txf	Created	Created container: storage-initializer
16:40:50	kserve-ci-e2e-test	multus	sklearn-v2-mlserver-predictor-7f568cdc6b-44txf	AddedInterface	Add eth0 [10.132.0.35/23] from ovn-kubernetes
16:40:55 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:40:55	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-44txf	Pulling	Pulling image "docker.io/seldonio/mlserver:1.7.1"
16:40:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	InferenceServiceReady	InferenceService [isvc-autogluon-v2] is Ready
16:41:06	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-ng6fr	Killing	Stopping container kserve-container
16:41:06	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-ng6fr	Killing	Stopping container kube-rbac-proxy
16:41:11	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-variants-predictor-5c889d8844	SuccessfulCreate	Created pod: isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2
16:41:11	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-variants-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-variants-predictor-5c889d8844 from 0 to 1
16:41:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2-variants": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-variants": the object has been modified; please apply your changes to the latest version and try again
16:41:12	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
16:41:13	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:41:13	kserve-ci-e2e-test	multus	isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2	AddedInterface	Add eth0 [10.133.0.30/23] from ovn-kubernetes
16:41:13	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2	Created	Created container: storage-initializer
16:41:13	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2	Started	Started container storage-initializer
16:41:18	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2	Started	Started container kube-rbac-proxy
16:41:18	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
16:41:18	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2	Created	Created container: kserve-container
16:41:18	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2	Started	Started container kserve-container
16:41:18	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:41:18	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2	Created	Created container: kube-rbac-proxy
16:41:56 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:41:56 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:42:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	InferenceServiceReady	InferenceService [isvc-autogluon-v2-variants] is Ready
16:42:54 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:42:56 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:42:56 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:43:03	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-44txf	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:43:03	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-44txf	Created	Created container: kserve-container
16:43:03	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-44txf	Started	Started container kserve-container
16:43:03	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-44txf	Created	Created container: kube-rbac-proxy
16:43:03	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-44txf	Started	Started container kube-rbac-proxy
16:43:03	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-44txf	Pulled	Successfully pulled image "docker.io/seldonio/mlserver:1.7.1" in 2m7.986s (2m7.986s including waiting). Image size: 10890461297 bytes.
16:43:05	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2	Unhealthy	Readiness probe failed: Get "http://10.133.0.30:8080/v2/models/isvc-autogluon-v2-variants/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
16:43:08	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2	Killing	Stopping container kube-rbac-proxy
16:43:08	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2	Killing	Stopping container kserve-container
16:43:10	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-llqg2	Unhealthy	Readiness probe failed: Get "https://10.133.0.30:8643/healthz": dial tcp 10.133.0.30:8643: connect: connection refused
16:43:14	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
16:43:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2-noslash": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-noslash": the object has been modified; please apply your changes to the latest version and try again
16:43:14	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-noslash-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-noslash-predictor-5499c79f9f from 0 to 1
16:43:14	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-noslash-predictor-5499c79f9f	SuccessfulCreate	Created pod: isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k
16:43:15	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k	Created	Created container: storage-initializer
16:43:15	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:43:15	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k	Started	Started container storage-initializer
16:43:15	kserve-ci-e2e-test	multus	isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k	AddedInterface	Add eth0 [10.133.0.31/23] from ovn-kubernetes
16:43:34 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:43:34 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:43:40	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-44txf	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
16:43:44	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k	Started	Started container kube-rbac-proxy
16:43:44	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
16:43:44	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k	Created	Created container: kserve-container
16:43:44	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k	Started	Started container kserve-container
16:43:44	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:43:44	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k	Created	Created container: kube-rbac-proxy
16:43:49	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:43:49	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:43:51	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	InferenceServiceReady	InferenceService [sklearn-v2-mlserver] is Ready
16:43:51 (x12)	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:43:59	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-44txf	Killing	Stopping container kube-rbac-proxy
16:43:59	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-44txf	Killing	Stopping container kserve-container
16:44:01	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-44txf	Unhealthy	Readiness probe failed: Get "http://10.132.0.35:8080/v2/models/sklearn-v2-mlserver/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
16:44:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-runtime": the object has been modified; please apply your changes to the latest version and try again
16:44:04	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-runtime-predictor-6fc7d5d5bd	SuccessfulCreate	Created pod: isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2
16:44:04	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-runtime-predictor-6fc7d5d5bd from 0 to 1
16:44:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2	Started	Started container storage-initializer
16:44:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2	Created	Created container: storage-initializer
16:44:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:44:05 (x2)	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-44txf	Unhealthy	Readiness probe failed: Get "https://10.132.0.35:8643/healthz": dial tcp 10.132.0.35:8643: connect: connection refused
16:44:05	kserve-ci-e2e-test	multus	isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2	AddedInterface	Add eth0 [10.133.0.32/23] from ovn-kubernetes
16:44:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine
16:44:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2	Created	Created container: kserve-container
16:44:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2	Started	Started container kube-rbac-proxy
16:44:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2	Created	Created container: kube-rbac-proxy
16:44:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:44:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2	Started	Started container kserve-container
16:44:14 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:44:14 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:44:19	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:44:19	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:44:21 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2	Unhealthy	Readiness probe failed: dial tcp 10.133.0.32:8080: connect: connection refused
16:44:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-runtime] is Ready
16:44:31 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:44:34	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:44:34	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:44:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2	Killing	Stopping container kube-rbac-proxy
16:44:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-6fc7d5d5bd-ckbh2	Killing	Stopping container kserve-container
16:44:42	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-runtime-predictor-5f69c7d498 from 0 to 1
16:44:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9	Started	Started container storage-initializer
16:44:42	kserve-ci-e2e-test	multus	isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9	AddedInterface	Add eth0 [10.132.0.36/23] from ovn-kubernetes
16:44:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:44:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9	Created	Created container: storage-initializer
16:44:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
16:44:42	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-runtime-predictor-5f69c7d498	SuccessfulCreate	Created pod: isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9
16:44:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9	Created	Created container: kube-rbac-proxy
16:44:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9	Started	Started container kserve-container
16:44:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9	Started	Started container kube-rbac-proxy
16:44:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9	Created	Created container: kserve-container
16:44:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:44:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
16:45:23	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	InferenceServiceReady	InferenceService [isvc-autogluon-v2-noslash] is Ready
16:45:23 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:45:25	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
16:45:27 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:45:27 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:45:29 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:45:29 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:45:30	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k	Killing	Stopping container kube-rbac-proxy
16:45:30	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-5ws9k	Killing	Stopping container kserve-container
16:45:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-ts-v1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1": the object has been modified; please apply your changes to the latest version and try again
16:45:35	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-ts-v1-predictor-6cf79d9f7d	SuccessfulCreate	Created pod: isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w
16:45:35	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-ts-v1-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-ts-v1-predictor-6cf79d9f7d from 0 to 1
16:45:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-v2-runtime] is Ready
16:45:35 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:45:36	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w	Created	Created container: storage-initializer
16:45:36	kserve-ci-e2e-test	multus	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w	AddedInterface	Add eth0 [10.133.0.33/23] from ovn-kubernetes
16:45:36	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:45:36	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w	Started	Started container storage-initializer
16:45:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9	Killing	Stopping container kserve-container
16:45:42	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:45:42	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:45:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9	Killing	Stopping container kube-rbac-proxy
16:45:45	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:45:45	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w	Started	Started container kserve-container
16:45:45	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w	Created	Created container: kserve-container
16:45:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9	Unhealthy	Readiness probe failed: Get "https://10.132.0.36:8643/healthz": dial tcp 10.132.0.36:8643: connect: connection refused
16:45:45	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w	Started	Started container kube-rbac-proxy
16:45:45	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w	Created	Created container: kube-rbac-proxy
16:45:45	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
16:45:47	kserve-ci-e2e-test	multus	isvc-sklearn-v2-predictor-ffc44684-v5xxc	AddedInterface	Add eth0 [10.133.0.34/23] from ovn-kubernetes
16:45:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
16:45:47	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-predictor-ffc44684 from 0 to 1
16:45:47	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-predictor-ffc44684	SuccessfulCreate	Created pod: isvc-sklearn-v2-predictor-ffc44684-v5xxc
16:45:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-ffc44684-v5xxc	Started	Started container storage-initializer
16:45:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-ffc44684-v5xxc	Created	Created container: storage-initializer
16:45:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-ffc44684-v5xxc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:45:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-ffc44684-v5xxc	Started	Started container kserve-container
16:45:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-ffc44684-v5xxc	Started	Started container kube-rbac-proxy
16:45:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-ffc44684-v5xxc	Created	Created container: kube-rbac-proxy
16:45:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-ffc44684-v5xxc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:45:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-ffc44684-v5xxc	Created	Created container: kserve-container
16:45:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-ffc44684-v5xxc	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine
16:45:53 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w	Unhealthy	Readiness probe failed: dial tcp 10.133.0.33:8080: connect: connection refused
16:46:03 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:46:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	InferenceServiceReady	InferenceService [isvc-autogluon-ts-v1] is Ready
16:46:05 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:46:05 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:46:18	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w	Killing	Stopping container kube-rbac-proxy
16:46:18	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w	Killing	Stopping container kserve-container
16:46:18	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-78r9w	Unhealthy	Readiness probe failed: Get "https://10.133.0.33:8643/healthz": dial tcp 10.133.0.33:8643: connect: connection refused
16:46:23	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-ts-v1-noslash-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-ts-v1-noslash-predictor-b54fbf585 from 0 to 1
16:46:23	kserve-ci-e2e-test	multus	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5	AddedInterface	Add eth0 [10.133.0.35/23] from ovn-kubernetes
16:46:23	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:46:23	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585	SuccessfulCreate	Created pod: isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5
16:46:23	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-ts-v1-noslash": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1-noslash": the object has been modified; please apply your changes to the latest version and try again
16:46:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5	Started	Started container storage-initializer
16:46:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5	Created	Created container: storage-initializer
16:46:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5	Started	Started container kube-rbac-proxy
16:46:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5	Created	Created container: kserve-container
16:46:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5	Started	Started container kserve-container
16:46:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:46:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5	Created	Created container: kube-rbac-proxy
16:46:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
16:46:32 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:46:32 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:46:36 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5	Unhealthy	Readiness probe failed: dial tcp 10.133.0.35:8080: connect: connection refused
16:46:38	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:46:38	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:46:46 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:46:46	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	InferenceServiceReady	InferenceService [isvc-autogluon-ts-v1-noslash] is Ready
16:46:53	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:46:53	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:46:58	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5	Killing	Stopping container kube-rbac-proxy
16:46:58	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-lnxb5	Killing	Stopping container kserve-container
16:47:03	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-6474b875b9	SuccessfulCreate	Created pod: isvc-canary-create-predictor-6474b875b9-zrzwc
16:47:03	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-6474b875b9	SuccessfulCreate	Created pod: isvc-canary-create-predictor-6474b875b9-l68gc
16:47:03 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again
16:47:03 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	UpdateFailed	Failed to update status for InferenceService "isvc-canary-create": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again
16:47:03	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-create-v2-predictor-96f77bdbd from 0 to 1
16:47:03	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-create-predictor-6474b875b9 from 0 to 2
16:47:03	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-v2-predictor-96f77bdbd	SuccessfulCreate	Created pod: isvc-canary-create-v2-predictor-96f77bdbd-v25rb
16:47:04	kserve-ci-e2e-test	multus	isvc-canary-create-v2-predictor-96f77bdbd-v25rb	AddedInterface	Add eth0 [10.133.0.36/23] from ovn-kubernetes
16:47:04	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-l68gc	Created	Created container: storage-initializer
16:47:04	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-l68gc	Started	Started container storage-initializer
16:47:04	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-zrzwc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:47:04	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-l68gc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:47:04	kserve-ci-e2e-test	multus	isvc-canary-create-predictor-6474b875b9-l68gc	AddedInterface	Add eth0 [10.133.0.37/23] from ovn-kubernetes
16:47:04	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-96f77bdbd-v25rb	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:47:04	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-96f77bdbd-v25rb	Created	Created container: storage-initializer
16:47:04	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-96f77bdbd-v25rb	Started	Started container storage-initializer
16:47:04	kserve-ci-e2e-test	multus	isvc-canary-create-predictor-6474b875b9-zrzwc	AddedInterface	Add eth0 [10.132.0.37/23] from ovn-kubernetes
16:47:04	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-zrzwc	Created	Created container: storage-initializer
16:47:04	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-zrzwc	Started	Started container storage-initializer
16:47:06 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-ffc44684-v5xxc	Unhealthy	Readiness probe failed: dial tcp 10.133.0.34:8080: connect: connection refused
16:47:08	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-96f77bdbd-v25rb	Started	Started container kube-rbac-proxy
16:47:08	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-96f77bdbd-v25rb	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:47:08	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-96f77bdbd-v25rb	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine
16:47:08	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-l68gc	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine
16:47:08	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-l68gc	Created	Created container: kserve-container
16:47:08	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-l68gc	Started	Started container kserve-container
16:47:08	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-96f77bdbd-v25rb	Created	Created container: kserve-container
16:47:08	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-96f77bdbd-v25rb	Created	Created container: kube-rbac-proxy
16:47:08	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-l68gc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:47:08	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-l68gc	Created	Created container: kube-rbac-proxy
16:47:08	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-l68gc	Started	Started container kube-rbac-proxy
16:47:08	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-96f77bdbd-v25rb	Started	Started container kserve-container
16:47:09	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-zrzwc	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928"
16:47:16 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:47:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	InferenceServiceReady	InferenceService [isvc-sklearn-v2] is Ready
16:47:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-ffc44684-v5xxc	Killing	Stopping container kube-rbac-proxy
16:47:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-ffc44684-v5xxc	Killing	Stopping container kserve-container
16:47:17 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:47:17 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:47:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-ffc44684-v5xxc	Unhealthy	Readiness probe failed: Get "https://10.133.0.34:8643/healthz": dial tcp 10.133.0.34:8643: connect: connection refused
16:47:22	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-zrzwc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:47:22	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-zrzwc	Started	Started container kserve-container
16:47:22	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-zrzwc	Created	Created container: kserve-container
16:47:22	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-zrzwc	Created	Created container: kube-rbac-proxy
16:47:22	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-zrzwc	Started	Started container kube-rbac-proxy
16:47:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-mixed": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-mixed": the object has been modified; please apply your changes to the latest version and try again
16:47:22	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-zrzwc	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" in 13.7s (13.7s including waiting). Image size: 1583654804 bytes.
16:47:22	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-mixed-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-mixed-predictor-89c454589 from 0 to 1
16:47:22	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-mixed-predictor-89c454589	SuccessfulCreate	Created pod: isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls
16:47:23	kserve-ci-e2e-test	multus	isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls	AddedInterface	Add eth0 [10.133.0.38/23] from ovn-kubernetes
16:47:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:47:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls	Created	Created container: storage-initializer
16:47:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls	Started	Started container storage-initializer
16:47:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls	Created	Created container: kserve-container
16:47:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls	Started	Started container kserve-container
16:47:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:47:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls	Created	Created container: kube-rbac-proxy
16:47:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine
16:47:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls	Started	Started container kube-rbac-proxy
16:47:37	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:47:37	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:48:20 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-l68gc	Unhealthy	Readiness probe failed: dial tcp 10.133.0.37:8080: connect: connection refused
16:48:26	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-l68gc	Killing	Stopping container kserve-container
16:48:26	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-6474b875b9	SuccessfulDelete	Deleted pod: isvc-canary-create-predictor-6474b875b9-l68gc
16:48:26	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-create-predictor-6474b875b9 from 2 to 1
16:48:26	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-l68gc	Killing	Stopping container kube-rbac-proxy
16:48:30 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-zrzwc	Unhealthy	Readiness probe failed: dial tcp 10.132.0.37:8080: connect: connection refused
16:48:30	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-l68gc	Unhealthy	Readiness probe failed: Get "https://10.133.0.37:8643/healthz": dial tcp 10.133.0.37:8643: connect: connection refused
16:48:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	InferenceServiceReady	InferenceService [isvc-canary-create] is Ready
16:48:40 (x17)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:48:43	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-96f77bdbd-v25rb	Killing	Stopping container kube-rbac-proxy
16:48:43	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-zrzwc	Killing	Stopping container kserve-container
16:48:43	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-96f77bdbd-v25rb	Killing	Stopping container kserve-container
16:48:43	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-zrzwc	Killing	Stopping container kube-rbac-proxy
16:48:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	InferenceServiceReady	InferenceService [isvc-sklearn-v2-mixed] is Ready
16:48:45 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:48:45	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-6474b875b9-zrzwc	Unhealthy	Readiness probe failed: Get "https://10.132.0.37:8643/healthz": dial tcp 10.132.0.37:8643: connect: connection refused
16:48:46 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-96f77bdbd-v25rb	Unhealthy	Readiness probe failed: dial tcp 10.133.0.36:8080: connect: connection refused
16:48:46	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-96f77bdbd-v25rb	Unhealthy	Readiness probe failed: Get "https://10.133.0.36:8643/healthz": dial tcp 10.133.0.36:8643: connect: connection refused
16:48:48	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-v2-predictor-5b4b466b66 from 0 to 1
16:48:49	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-l6qgw	Created	Created container: storage-initializer
16:48:49	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-s5xm7	Created	Created container: storage-initializer
16:48:49	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
16:48:49	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-predictor-7cf78f884f from 0 to 2
16:48:49	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-7cf78f884f	SuccessfulCreate	Created pod: isvc-canary-promote-predictor-7cf78f884f-l6qgw
16:48:49	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-7cf78f884f	SuccessfulCreate	Created pod: isvc-canary-promote-predictor-7cf78f884f-s5xm7
16:48:49	kserve-ci-e2e-test	multus	isvc-canary-promote-predictor-7cf78f884f-l6qgw	AddedInterface	Add eth0 [10.132.0.38/23] from ovn-kubernetes
16:48:49	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-l6qgw	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:48:49	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-l6qgw	Started	Started container storage-initializer
16:48:49	kserve-ci-e2e-test	multus	isvc-canary-promote-predictor-7cf78f884f-s5xm7	AddedInterface	Add eth0 [10.133.0.40/23] from ovn-kubernetes
16:48:49	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-s5xm7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:48:49	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-v2-predictor-5b4b466b66	SuccessfulCreate	Created pod: isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h
16:48:49	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-s5xm7	Started	Started container storage-initializer
16:48:50	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h	Created	Created container: storage-initializer
16:48:50	kserve-ci-e2e-test	multus	isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h	AddedInterface	Add eth0 [10.133.0.39/23] from ovn-kubernetes
16:48:50	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:48:50	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h	Started	Started container storage-initializer
16:48:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls	Killing	Stopping container kserve-container
16:48:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls	Killing	Stopping container kube-rbac-proxy
16:48:52 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:48:52 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:48:54	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-l6qgw	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:48:54	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-s5xm7	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:48:54	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h	Created	Created container: kserve-container
16:48:54	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h	Started	Started container kserve-container
16:48:54	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:48:54	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h	Created	Created container: kube-rbac-proxy
16:48:54	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h	Started	Started container kube-rbac-proxy
16:48:54	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-s5xm7	Started	Started container kube-rbac-proxy
16:48:54	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-s5xm7	Created	Created container: kube-rbac-proxy
16:48:54	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-l6qgw	Started	Started container kserve-container
16:48:54	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-s5xm7	Started	Started container kserve-container
16:48:54	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-s5xm7	Created	Created container: kserve-container
16:48:54	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-s5xm7	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine
16:48:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls	Unhealthy	Readiness probe failed: Get "https://10.133.0.38:8643/healthz": dial tcp 10.133.0.38:8643: connect: connection refused
16:48:54	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-l6qgw	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine
16:48:54	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-l6qgw	Created	Created container: kserve-container
16:48:54	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine
16:48:54 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-89c454589-tw2ls	Unhealthy	Readiness probe failed: dial tcp 10.133.0.38:8080: connect: connection refused
16:48:54	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-l6qgw	Started	Started container kube-rbac-proxy
16:48:54	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-l6qgw	Created	Created container: kube-rbac-proxy
16:48:58	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-lf8vp	Created	Created container: storage-initializer
16:48:58	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-predictor-547dfc5669	SuccessfulCreate	Created pod: isvc-tensorflow-predictor-547dfc5669-lf8vp
16:48:58	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow": the object has been modified; please apply your changes to the latest version and try again
16:48:58	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-lf8vp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:48:58	kserve-ci-e2e-test	multus	isvc-tensorflow-predictor-547dfc5669-lf8vp	AddedInterface	Add eth0 [10.133.0.41/23] from ovn-kubernetes
16:48:58	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-predictor-547dfc5669 from 0 to 1
16:48:58	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-lf8vp	Started	Started container storage-initializer
16:49:04	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-lf8vp	Pulling	Pulling image "tensorflow/serving:2.6.2"
16:49:08	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-lf8vp	Started	Started container kserve-container
16:49:08	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-lf8vp	Created	Created container: kube-rbac-proxy
16:49:08	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-lf8vp	Started	Started container kube-rbac-proxy
16:49:08	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-lf8vp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:49:08	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-lf8vp	Created	Created container: kserve-container
16:49:08	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-lf8vp	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 3.755s (3.755s including waiting). Image size: 425873876 bytes.
16:49:15 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-lf8vp	Unhealthy	Readiness probe failed: dial tcp 10.133.0.41:8080: connect: connection refused
16:49:25 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:49:25	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	InferenceServiceReady	InferenceService [isvc-tensorflow] is Ready
16:49:28 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:49:28 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:49:39	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-lf8vp	Killing	Stopping container kserve-container
16:49:39	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-lf8vp	Killing	Stopping container kube-rbac-proxy
16:49:44	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-runtime-predictor-855cfcd677	SuccessfulCreate	Created pod: isvc-tensorflow-runtime-predictor-855cfcd677-6fp45
16:49:44	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-runtime-predictor-855cfcd677 from 0 to 1
16:49:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow-runtime": the object has been modified; please apply your changes to the latest version and try again
16:49:45	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-6fp45	Created	Created container: storage-initializer
16:49:45	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-6fp45	Started	Started container storage-initializer
16:49:45	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-6fp45	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:49:45	kserve-ci-e2e-test	multus	isvc-tensorflow-runtime-predictor-855cfcd677-6fp45	AddedInterface	Add eth0 [10.132.0.39/23] from ovn-kubernetes
16:49:50	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-6fp45	Pulling	Pulling image "tensorflow/serving:2.6.2"
16:49:54	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-6fp45	Started	Started container kube-rbac-proxy
16:49:54	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-6fp45	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 3.873s (3.873s including waiting). Image size: 425873876 bytes.
16:49:54	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-6fp45	Created	Created container: kserve-container
16:49:54	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-6fp45	Started	Started container kserve-container
16:49:54	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-6fp45	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:49:54	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-6fp45	Created	Created container: kube-rbac-proxy
16:50:00 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-6fp45	Unhealthy	Readiness probe failed: dial tcp 10.132.0.39:8080: connect: connection refused
16:50:01 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-l6qgw	Unhealthy	Readiness probe failed: dial tcp 10.132.0.38:8080: connect: connection refused
16:50:05 (x6)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-lf8vp	Unhealthy	Readiness probe failed: Get "https://10.133.0.41:8643/healthz": dial tcp 10.133.0.41:8643: connect: connection refused
16:50:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	InferenceServiceReady	InferenceService [isvc-tensorflow-runtime] is Ready
16:50:11 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:50:14 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:50:14 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:50:21	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-promote-predictor-7cf78f884f from 2 to 1
16:50:21 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again
16:50:21	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-7cf78f884f	SuccessfulDelete	Deleted pod: isvc-canary-promote-predictor-7cf78f884f-s5xm7
16:50:21	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-s5xm7	Killing	Stopping container kube-rbac-proxy
16:50:21	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-s5xm7	Killing	Stopping container kserve-container
16:50:21 (x21)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:50:21 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	UpdateFailed	Failed to update status for InferenceService "isvc-canary-promote": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again
16:50:22	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-s5xm7	Unhealthy	Readiness probe failed: Get "https://10.133.0.40:8643/healthz": dial tcp 10.133.0.40:8643: connect: connection refused
16:50:22 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-s5xm7	Unhealthy	Readiness probe failed: dial tcp 10.133.0.40:8080: connect: connection refused
16:50:24	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-l6qgw	Killing	Stopping container kube-rbac-proxy
16:50:24	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-8v72b	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:50:24	kserve-ci-e2e-test	multus	isvc-canary-promote-v2-predictor-5b4b466b66-8v72b	AddedInterface	Add eth0 [10.132.0.40/23] from ovn-kubernetes
16:50:24	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-v2-predictor-5b4b466b66	SuccessfulCreate	Created pod: isvc-canary-promote-v2-predictor-5b4b466b66-8v72b
16:50:24	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-v2-predictor-5b4b466b66 from 1 to 2
16:50:24	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-l6qgw	Killing	Stopping container kserve-container
16:50:25	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-8v72b	Created	Created container: storage-initializer
16:50:25	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-8v72b	Started	Started container storage-initializer
16:50:26	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-7cf78f884f-l6qgw	Unhealthy	Readiness probe failed: Get "https://10.132.0.38:8643/healthz": dial tcp 10.132.0.38:8643: connect: connection refused
16:50:26	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-6fp45	Killing	Stopping container kserve-container
16:50:26	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-6fp45	Killing	Stopping container kube-rbac-proxy
16:50:29	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-8v72b	Started	Started container kube-rbac-proxy
16:50:29	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-8v72b	Created	Created container: kserve-container
16:50:29	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-8v72b	Started	Started container kserve-container
16:50:29	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-8v72b	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:50:29	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-8v72b	Created	Created container: kube-rbac-proxy
16:50:29	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-8v72b	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine
16:50:31	kserve-ci-e2e-test	deployment-controller	isvc-triton-predictor	ScalingReplicaSet	Scaled up replica set isvc-triton-predictor-5d7db58676 from 0 to 1
16:50:31	kserve-ci-e2e-test	replicaset-controller	isvc-triton-predictor-5d7db58676	SuccessfulCreate	Created pod: isvc-triton-predictor-5d7db58676-k4mg9
16:50:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	UpdateFailed	Failed to update status for InferenceService "isvc-triton": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-triton": the object has been modified; please apply your changes to the latest version and try again
16:50:32	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-k4mg9	Started	Started container storage-initializer
16:50:32	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-k4mg9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:50:32	kserve-ci-e2e-test	multus	isvc-triton-predictor-5d7db58676-k4mg9	AddedInterface	Add eth0 [10.133.0.42/23] from ovn-kubernetes
16:50:32	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-k4mg9	Created	Created container: storage-initializer
16:50:36	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-k4mg9	Pulling	Pulling image "nvcr.io/nvidia/tritonserver:23.05-py3"
16:50:55 (x6)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-6fp45	Unhealthy	Readiness probe failed: Get "https://10.132.0.39:8643/healthz": dial tcp 10.132.0.39:8643: connect: connection refused
16:51:46 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	InferenceServiceReady	InferenceService [isvc-canary-promote] is Ready
16:51:54	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h	Killing	Stopping container kube-rbac-proxy
16:51:54	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h	Killing	Stopping container kserve-container
16:51:54	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-8v72b	Killing	Stopping container kserve-container
16:51:54	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-8v72b	Killing	Stopping container kube-rbac-proxy
16:51:56 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-8v72b	Unhealthy	Readiness probe failed: dial tcp 10.132.0.40:8080: connect: connection refused
16:51:56	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-8v72b	Unhealthy	Readiness probe failed: Get "https://10.132.0.40:8643/healthz": dial tcp 10.132.0.40:8643: connect: connection refused
16:51:59	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-v2-predictor-7b8b964c7b	SuccessfulCreate	Created pod: isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj
16:51:59	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-6b8c75c7b9	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk
16:51:59 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again
16:51:59	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-predictor-serving-cert" not found
16:51:59	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-predictor-6b8c75c7b9 from 0 to 2
16:51:59	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-6b8c75c7b9	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk
16:51:59	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-v2-predictor-7b8b964c7b from 0 to 1
16:52:00	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk	Started	Started container storage-initializer
16:52:00	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk	AddedInterface	Add eth0 [10.132.0.41/23] from ovn-kubernetes
16:52:00	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:52:00	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk	Created	Created container: storage-initializer
16:52:01 (x2)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h	Unhealthy	Readiness probe failed: Get "https://10.133.0.39:8643/healthz": dial tcp 10.133.0.39:8643: connect: connection refused
16:52:01 (x11)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5b4b466b66-g7b8h	Unhealthy	Readiness probe failed: dial tcp 10.133.0.39:8080: connect: connection refused
16:52:02	kserve-ci-e2e-test	multus	isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj	AddedInterface	Add eth0 [10.133.0.43/23] from ovn-kubernetes
16:52:02	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk	AddedInterface	Add eth0 [10.133.0.44/23] from ovn-kubernetes
16:52:04	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk	Created	Created container: kube-rbac-proxy
16:52:04	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk	Started	Started container kube-rbac-proxy
16:52:04	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:52:04	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk	Started	Started container kserve-container
16:52:04	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk	Created	Created container: kserve-container
16:52:04	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine
16:52:30	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk	Created	Created container: storage-initializer
16:52:30	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk	Started	Started container storage-initializer
16:52:30	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-k4mg9	Pulled	Successfully pulled image "nvcr.io/nvidia/tritonserver:23.05-py3" in 1m54.253s (1m54.253s including waiting). Image size: 12907074623 bytes.
16:52:30	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:52:30	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:52:30	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj	Started	Started container storage-initializer
16:52:30	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj	Created	Created container: storage-initializer
16:52:31	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-k4mg9	Started	Started container kserve-container
16:52:31 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:52:31 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:52:31	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-k4mg9	Created	Created container: kserve-container
16:52:31	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-k4mg9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:52:31	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-k4mg9	Created	Created container: kube-rbac-proxy
16:52:31	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-k4mg9	Started	Started container kube-rbac-proxy
16:52:33 (x2)	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-k4mg9	Unhealthy	Readiness probe failed: dial tcp 10.133.0.42:8080: connect: connection refused
16:52:35	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:52:35	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk	Created	Created container: kserve-container
16:52:35	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk	Created	Created container: kube-rbac-proxy
16:52:35	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine
16:52:35	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine
16:52:35	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj	Created	Created container: kserve-container
16:52:35	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj	Started	Started container kserve-container
16:52:35	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:52:35	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj	Created	Created container: kube-rbac-proxy
16:52:35	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj	Started	Started container kube-rbac-proxy
16:52:35	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk	Started	Started container kube-rbac-proxy
16:52:35	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk	Started	Started container kserve-container
16:52:38 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:52:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	InferenceServiceReady	InferenceService [isvc-triton] is Ready
16:52:45	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-k4mg9	Killing	Stopping container kube-rbac-proxy
16:52:45	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-k4mg9	Killing	Stopping container kserve-container
16:52:50	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-predictor-7566d65bf7	SuccessfulCreate	Created pod: isvc-xgboost-predictor-7566d65bf7-cmnxd
16:52:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost": the object has been modified; please apply your changes to the latest version and try again
16:52:50	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-predictor-7566d65bf7 from 0 to 1
16:52:51	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-cmnxd	Created	Created container: storage-initializer
16:52:51	kserve-ci-e2e-test	multus	isvc-xgboost-predictor-7566d65bf7-cmnxd	AddedInterface	Add eth0 [10.132.0.42/23] from ovn-kubernetes
16:52:51	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-cmnxd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:52:51	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-cmnxd	Started	Started container storage-initializer
16:52:55	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-cmnxd	Pulling	Pulling image "kserve/xgbserver:latest"
16:53:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-cmnxd	Created	Created container: kserve-container
16:53:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-cmnxd	Created	Created container: kube-rbac-proxy
16:53:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-cmnxd	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 17.294s (17.294s including waiting). Image size: 1332343041 bytes.
16:53:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-cmnxd	Started	Started container kserve-container
16:53:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-cmnxd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:53:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-cmnxd	Started	Started container kube-rbac-proxy
16:53:43 (x9)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk	Unhealthy	Readiness probe failed: dial tcp 10.133.0.44:8080: connect: connection refused
16:53:50 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:53:50 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:53:53 (x20)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:53:53 (x3)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	UpdateFailed	Failed to update status for InferenceService "isvc-canary-rollback": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again
16:53:53	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-6b8c75c7b9	SuccessfulDelete	Deleted pod: isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk
16:53:53	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-rollback-predictor-6b8c75c7b9 from 2 to 1
16:53:54	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk	Killing	Stopping container kube-rbac-proxy
16:53:54	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk	Killing	Stopping container kserve-container
16:53:58	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-wjkpk	Unhealthy	Readiness probe failed: Get "https://10.133.0.44:8643/healthz": dial tcp 10.133.0.44:8643: connect: connection refused
16:53:59	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj	Killing	Stopping container kserve-container
16:53:59	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-6b8c75c7b9	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-6b8c75c7b9-s77ll
16:53:59	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj	Killing	Stopping container kube-rbac-proxy
16:53:59	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-predictor-6b8c75c7b9 from 1 to 2
16:54:00	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-6b8c75c7b9-s77ll	AddedInterface	Add eth0 [10.133.0.45/23] from ovn-kubernetes
16:54:00	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-s77ll	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:54:00	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-s77ll	Created	Created container: storage-initializer
16:54:00	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-s77ll	Started	Started container storage-initializer
16:54:03 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj	Unhealthy	Readiness probe failed: dial tcp 10.133.0.43:8080: connect: connection refused
16:54:03	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-7b8b964c7b-c8lhj	Unhealthy	Readiness probe failed: Get "https://10.133.0.43:8643/healthz": dial tcp 10.133.0.43:8643: connect: connection refused
16:54:04	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-s77ll	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine
16:54:04	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-s77ll	Started	Started container kube-rbac-proxy
16:54:04	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-s77ll	Created	Created container: kube-rbac-proxy
16:54:04	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-s77ll	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:54:04	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-s77ll	Started	Started container kserve-container
16:54:04	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-s77ll	Created	Created container: kserve-container
16:54:05 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:54:05 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:54:09 (x8)	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-cmnxd	Unhealthy	Readiness probe failed: dial tcp 10.132.0.42:8080: connect: connection refused
16:54:20	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-cmnxd	Killing	Stopping container kube-rbac-proxy
16:54:20	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-cmnxd	Killing	Stopping container kserve-container
16:54:20 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:54:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	InferenceServiceReady	InferenceService [isvc-xgboost] is Ready
16:54:25	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-mlserver-predictor-7b55b84f54	SuccessfulCreate	Created pod: isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q
16:54:25	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-mlserver-predictor-7b55b84f54 from 0 to 1
16:54:25	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
16:54:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q	Created	Created container: storage-initializer
16:54:26	kserve-ci-e2e-test	multus	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q	AddedInterface	Add eth0 [10.132.0.43/23] from ovn-kubernetes
16:54:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:54:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q	Started	Started container storage-initializer
16:54:31	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:54:31	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q	Started	Started container kube-rbac-proxy
16:54:31	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q	Created	Created container: kube-rbac-proxy
16:54:31	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q	Started	Started container kserve-container
16:54:31	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q	Created	Created container: kserve-container
16:54:31	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
16:54:55	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:54:55	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:55:08 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:55:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	InferenceServiceReady	InferenceService [isvc-xgboost-v2-mlserver] is Ready
16:55:10 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:55:10 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:55:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q	Killing	Stopping container kube-rbac-proxy
16:55:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q	Killing	Stopping container kserve-container
16:55:18	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q	Unhealthy	Readiness probe failed: Get "https://10.132.0.43:8643/healthz": dial tcp 10.132.0.43:8643: connect: connection refused
16:55:21	kserve-ci-e2e-test	replicaset-controller	xgboost-v2-mlserver-predictor-59944bcf4c	SuccessfulCreate	Created pod: xgboost-v2-mlserver-predictor-59944bcf4c-kn69w
16:55:21	kserve-ci-e2e-test	multus	xgboost-v2-mlserver-predictor-59944bcf4c-kn69w	AddedInterface	Add eth0 [10.132.0.44/23] from ovn-kubernetes
16:55:21	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-kn69w	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:55:21	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-kn69w	Created	Created container: storage-initializer
16:55:21	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-kn69w	Started	Started container storage-initializer
16:55:21	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
16:55:21	kserve-ci-e2e-test	deployment-controller	xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set xgboost-v2-mlserver-predictor-59944bcf4c from 0 to 1
16:55:22 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	InferenceServiceReady	InferenceService [isvc-canary-rollback] is Ready
16:55:26	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-kn69w	Started	Started container kube-rbac-proxy
16:55:26	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-kn69w	Created	Created container: kube-rbac-proxy
16:55:26	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-kn69w	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
16:55:26	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-kn69w	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:55:26	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-kn69w	Started	Started container kserve-container
16:55:26	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-kn69w	Created	Created container: kserve-container
16:55:29	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk	Killing	Stopping container kube-rbac-proxy
16:55:29	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk	Killing	Stopping container kserve-container
16:55:29	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-s77ll	Killing	Stopping container kserve-container
16:55:29	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-s77ll	Killing	Stopping container kube-rbac-proxy
16:55:31	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk	Unhealthy	Readiness probe failed: Get "https://10.132.0.41:8643/healthz": dial tcp 10.132.0.41:8643: connect: connection refused
16:55:31 (x11)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk	Unhealthy	Readiness probe failed: dial tcp 10.132.0.41:8080: connect: connection refused
16:55:32 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-s77ll	Unhealthy	Readiness probe failed: dial tcp 10.133.0.45:8080: connect: connection refused
16:55:32	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6b8c75c7b9-s77ll	Unhealthy	Readiness probe failed: Get "https://10.133.0.45:8643/healthz": dial tcp 10.133.0.45:8643: connect: connection refused
16:55:34	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-stop-v2-predictor-596664c988 from 0 to 1
16:55:35	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-5cdcf6545	SuccessfulCreate	Created pod: isvc-canary-stop-predictor-5cdcf6545-mth9z
16:55:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-596664c988-bsz9b	Started	Started container storage-initializer
16:55:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-kqh6q	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-stop-predictor-serving-cert" not found
16:55:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-mth9z	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-stop-predictor-serving-cert" not found
16:55:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	UpdateFailed	Failed to update status for InferenceService "isvc-canary-stop": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again
16:55:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again
16:55:35	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-v2-predictor-596664c988	SuccessfulCreate	Created pod: isvc-canary-stop-v2-predictor-596664c988-bsz9b
16:55:35	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-5cdcf6545	SuccessfulCreate	Created pod: isvc-canary-stop-predictor-5cdcf6545-kqh6q
16:55:35	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-stop-predictor-5cdcf6545 from 0 to 2
16:55:35	kserve-ci-e2e-test	multus	isvc-canary-stop-v2-predictor-596664c988-bsz9b	AddedInterface	Add eth0 [10.133.0.46/23] from ovn-kubernetes
16:55:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-596664c988-bsz9b	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:55:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-596664c988-bsz9b	Created	Created container: storage-initializer
16:55:36	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-kqh6q	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:55:36	kserve-ci-e2e-test	multus	isvc-canary-stop-predictor-5cdcf6545-kqh6q	AddedInterface	Add eth0 [10.133.0.47/23] from ovn-kubernetes
16:55:36	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:55:36	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-kqh6q	Created	Created container: storage-initializer
16:55:36	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-kqh6q	Started	Started container storage-initializer
16:55:36	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-mth9z	Started	Started container storage-initializer
16:55:36	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-mth9z	Created	Created container: storage-initializer
16:55:36	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-mth9z	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:55:36	kserve-ci-e2e-test	multus	isvc-canary-stop-predictor-5cdcf6545-mth9z	AddedInterface	Add eth0 [10.132.0.45/23] from ovn-kubernetes
16:55:36	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:55:40	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-596664c988-bsz9b	Created	Created container: kserve-container
16:55:40	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-596664c988-bsz9b	Started	Started container kserve-container
16:55:40	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-596664c988-bsz9b	Started	Started container kube-rbac-proxy
16:55:40	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-596664c988-bsz9b	Created	Created container: kube-rbac-proxy
16:55:40	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-596664c988-bsz9b	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:55:40	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-mth9z	Started	Started container kube-rbac-proxy
16:55:40	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-mth9z	Created	Created container: kube-rbac-proxy
16:55:40	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-mth9z	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:55:40	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-mth9z	Started	Started container kserve-container
16:55:40	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-mth9z	Created	Created container: kserve-container
16:55:40	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-mth9z	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine
16:55:40	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-596664c988-bsz9b	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine
16:55:41	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-kqh6q	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine
16:55:41	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-kqh6q	Created	Created container: kube-rbac-proxy
16:55:41	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-kqh6q	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:55:41	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-kqh6q	Started	Started container kserve-container
16:55:41	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-kqh6q	Created	Created container: kserve-container
16:55:41	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-kqh6q	Started	Started container kube-rbac-proxy
16:56:04	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	InferenceServiceReady	InferenceService [xgboost-v2-mlserver] is Ready
16:56:04 (x10)	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:56:06 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:56:06 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:56:11	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-kn69w	Killing	Stopping container kube-rbac-proxy
16:56:11	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-kn69w	Killing	Stopping container kserve-container
16:56:14	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-kn69w	Unhealthy	Readiness probe failed: Get "http://10.132.0.44:8080/v2/models/xgboost-v2-mlserver/ready": dial tcp 10.132.0.44:8080: connect: connection refused
16:56:14	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-kn69w	Unhealthy	Readiness probe failed: Get "https://10.132.0.44:8643/healthz": dial tcp 10.132.0.44:8643: connect: connection refused
16:56:16	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-runtime-predictor-54dd488c85 from 0 to 1
16:56:16	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-runtime-predictor-54dd488c85	SuccessfulCreate	Created pod: isvc-xgboost-runtime-predictor-54dd488c85-v47xd
16:56:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-runtime": the object has been modified; please apply your changes to the latest version and try again
16:56:17	kserve-ci-e2e-test	multus	isvc-xgboost-runtime-predictor-54dd488c85-v47xd	AddedInterface	Add eth0 [10.133.0.48/23] from ovn-kubernetes
16:56:17	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-v47xd	Created	Created container: storage-initializer
16:56:17	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-v47xd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:56:17	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-v47xd	Started	Started container storage-initializer
16:56:21	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-v47xd	Pulling	Pulling image "kserve/xgbserver:latest"
16:56:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-v47xd	Created	Created container: kserve-container
16:56:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-v47xd	Started	Started container kube-rbac-proxy
16:56:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-v47xd	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 18.407s (18.407s including waiting). Image size: 1332343041 bytes.
16:56:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-v47xd	Started	Started container kserve-container
16:56:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-v47xd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:56:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-v47xd	Created	Created container: kube-rbac-proxy
16:56:49 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-mth9z	Unhealthy	Readiness probe failed: dial tcp 10.132.0.45:8080: connect: connection refused
16:56:58 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-596664c988-bsz9b	Unhealthy	Readiness probe failed: dial tcp 10.133.0.46:8080: connect: connection refused
16:56:59	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	InferenceServiceReady	InferenceService [isvc-canary-stop] is Ready
16:57:01 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:57:01 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:57:08	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-kqh6q	Killing	Stopping container kserve-container
16:57:08	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-predictor	ScalingReplicaSet	Scaled down replica set isvc-canary-stop-predictor-5cdcf6545 from 2 to 1
16:57:08	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-5cdcf6545	SuccessfulDelete	Deleted pod: isvc-canary-stop-predictor-5cdcf6545-kqh6q
16:57:08	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-kqh6q	Killing	Stopping container kube-rbac-proxy
16:57:09 (x11)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-kqh6q	Unhealthy	Readiness probe failed: dial tcp 10.133.0.47:8080: connect: connection refused
16:57:09	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-kqh6q	Unhealthy	Readiness probe failed: Get "https://10.133.0.47:8643/healthz": dial tcp 10.133.0.47:8643: connect: connection refused
16:57:10	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-mth9z	Killing	Stopping container kserve-container
16:57:10	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-mth9z	Killing	Stopping container kube-rbac-proxy
16:57:10	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-596664c988-bsz9b	Killing	Stopping container kserve-container
16:57:10	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-596664c988-bsz9b	Killing	Stopping container kube-rbac-proxy
16:57:10 (x23)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:57:13	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-596664c988-bsz9b	Unhealthy	Readiness probe failed: Get "https://10.133.0.46:8643/healthz": dial tcp 10.133.0.46:8643: connect: connection refused
16:57:13	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-5cdcf6545-mth9z	Unhealthy	Readiness probe failed: Get "https://10.132.0.45:8643/healthz": dial tcp 10.132.0.45:8643: connect: connection refused
16:57:20	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nz97x	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-lightgbm-predictor-serving-cert" not found
16:57:20 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm": the object has been modified; please apply your changes to the latest version and try again
16:57:20	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-predictor-54f85ddf88	SuccessfulCreate	Created pod: isvc-lightgbm-predictor-54f85ddf88-nz97x
16:57:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm": the object has been modified; please apply your changes to the latest version and try again
16:57:20	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-predictor-54f85ddf88 from 0 to 1
16:57:21	kserve-ci-e2e-test	multus	isvc-lightgbm-predictor-54f85ddf88-nz97x	AddedInterface	Add eth0 [10.133.0.49/23] from ovn-kubernetes
16:57:21	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nz97x	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:57:21	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nz97x	Created	Created container: storage-initializer
16:57:21	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nz97x	Started	Started container storage-initializer
16:57:25	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nz97x	Pulling	Pulling image "kserve/lgbserver:latest"
16:57:31	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nz97x	Pulled	Successfully pulled image "kserve/lgbserver:latest" in 6.381s (6.381s including waiting). Image size: 632170260 bytes.
16:57:31	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nz97x	Started	Started container kserve-container
16:57:31	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nz97x	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:57:31	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nz97x	Created	Created container: kserve-container
16:57:32	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nz97x	Created	Created container: kube-rbac-proxy
16:57:32	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nz97x	Started	Started container kube-rbac-proxy
16:57:46 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:57:46 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:57:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-runtime] is Ready
16:57:47 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:57:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-v47xd	Killing	Stopping container kube-rbac-proxy
16:57:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-v47xd	Killing	Stopping container kserve-container
16:57:57	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-v47xd	Unhealthy	Readiness probe failed: Get "https://10.133.0.48:8643/healthz": dial tcp 10.133.0.48:8643: connect: connection refused
16:57:57 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-v47xd	Unhealthy	Readiness probe failed: dial tcp 10.133.0.48:8080: connect: connection refused
16:58:01	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-runtime-predictor-7787c55dc9	SuccessfulCreate	Created pod: isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm
16:58:01	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-runtime-predictor-7787c55dc9 from 0 to 1
16:58:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
16:58:02	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm	Started	Started container storage-initializer
16:58:02	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm	Created	Created container: storage-initializer
16:58:02	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:58:02	kserve-ci-e2e-test	multus	isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm	AddedInterface	Add eth0 [10.132.0.46/23] from ovn-kubernetes
16:58:05 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:58:05 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:58:07	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm	Started	Started container kserve-container
16:58:07	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm	Started	Started container kube-rbac-proxy
16:58:07	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
16:58:07	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm	Created	Created container: kserve-container
16:58:07	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:58:07	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm	Created	Created container: kube-rbac-proxy
16:58:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:58:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:58:45	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
16:58:46	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:58:46	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:58:50 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:58:50 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:58:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-v2-runtime] is Ready
16:58:55 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:58:55 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:58:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	InferenceServiceReady	InferenceService [isvc-lightgbm] is Ready
16:59:01	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm	Killing	Stopping container kserve-container
16:59:01	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm	Killing	Stopping container kube-rbac-proxy
16:59:04	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nz97x	Killing	Stopping container kube-rbac-proxy
16:59:04	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nz97x	Killing	Stopping container kserve-container
16:59:05	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm	Unhealthy	Readiness probe failed: Get "http://10.132.0.46:8080/v2/models/isvc-xgboost-v2-runtime/ready": dial tcp 10.132.0.46:8080: connect: connection refused
16:59:05	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm	Unhealthy	Readiness probe failed: Get "https://10.132.0.46:8643/healthz": dial tcp 10.132.0.46:8643: connect: connection refused
16:59:05 (x11)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nz97x	Unhealthy	Readiness probe failed: dial tcp 10.133.0.49:8080: connect: connection refused
16:59:06	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-predictor-fcc668bc from 0 to 1
16:59:07	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-predictor-fcc668bc	SuccessfulCreate	Created pod: isvc-xgboost-v2-predictor-fcc668bc-nmnxm
16:59:07	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
16:59:07	kserve-ci-e2e-test	multus	isvc-xgboost-v2-predictor-fcc668bc-nmnxm	AddedInterface	Add eth0 [10.133.0.50/23] from ovn-kubernetes
16:59:07	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-nmnxm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:59:07	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-nmnxm	Created	Created container: storage-initializer
16:59:07	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-nmnxm	Started	Started container storage-initializer
16:59:07 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
16:59:09	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-runtime-predictor-577f7db7cb	SuccessfulCreate	Created pod: isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd
16:59:09	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-runtime-predictor-577f7db7cb from 0 to 1
16:59:09	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-runtime": the object has been modified; please apply your changes to the latest version and try again
16:59:09	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-nz97x	Unhealthy	Readiness probe failed: Get "https://10.133.0.49:8643/healthz": dial tcp 10.133.0.49:8643: connect: connection refused
16:59:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
16:59:10	kserve-ci-e2e-test	multus	isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd	AddedInterface	Add eth0 [10.133.0.51/23] from ovn-kubernetes
16:59:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-nmnxm	Created	Created container: kserve-container
16:59:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-nmnxm	Started	Started container kserve-container
16:59:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-nmnxm	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:59:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-nmnxm	Pulled	Container image "kserve/xgbserver:latest" already present on machine
16:59:11	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
16:59:11	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd	Created	Created container: storage-initializer
16:59:11	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd	Started	Started container storage-initializer
16:59:12	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-nmnxm	Created	Created container: kube-rbac-proxy
16:59:12	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-nmnxm	Started	Started container kube-rbac-proxy
16:59:15	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd	Pulled	Container image "kserve/lgbserver:latest" already present on machine
16:59:16	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd	Started	Started container kube-rbac-proxy
16:59:16	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd	Created	Created container: kserve-container
16:59:16	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd	Started	Started container kserve-container
16:59:16	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd	Created	Created container: kube-rbac-proxy
16:59:16	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:59:52 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:59:52 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:59:54 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:59:54 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:00:19 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:00:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	InferenceServiceReady	InferenceService [isvc-xgboost-v2] is Ready
17:00:22 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:00:22 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:00:27	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-nmnxm	Killing	Stopping container kube-rbac-proxy
17:00:27	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-nmnxm	Killing	Stopping container kserve-container
17:00:28	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-nmnxm	Unhealthy	Readiness probe failed: Get "https://10.133.0.50:8643/healthz": dial tcp 10.133.0.50:8643: connect: connection refused
17:00:28 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-nmnxm	Unhealthy	Readiness probe failed: dial tcp 10.133.0.50:8080: connect: connection refused
17:00:29 (x10)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd	Unhealthy	Readiness probe failed: dial tcp 10.133.0.51:8080: connect: connection refused
17:00:32	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3": the object has been modified; please apply your changes to the latest version and try again
17:00:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-84d46f45-8fxg6	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-predictor-serving-cert" not found
17:00:32	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-predictor-84d46f45	SuccessfulCreate	Created pod: isvc-sklearn-s3-predictor-84d46f45-8fxg6
17:00:32	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-predictor-84d46f45 from 0 to 1
17:00:33	kserve-ci-e2e-test	multus	isvc-sklearn-s3-predictor-84d46f45-8fxg6	AddedInterface	Add eth0 [10.133.0.52/23] from ovn-kubernetes
17:00:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-84d46f45-8fxg6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
17:00:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-84d46f45-8fxg6	Created	Created container: storage-initializer
17:00:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-84d46f45-8fxg6	Started	Started container storage-initializer
17:00:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-84d46f45-8fxg6	Started	Started container kserve-container
17:00:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-84d46f45-8fxg6	Created	Created container: kserve-container
17:00:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-84d46f45-8fxg6	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:00:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-84d46f45-8fxg6	Created	Created container: kube-rbac-proxy
17:00:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-84d46f45-8fxg6	Started	Started container kube-rbac-proxy
17:00:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-84d46f45-8fxg6	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine
17:00:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-runtime] is Ready
17:00:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:00:39 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:00:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:00:50	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd	Killing	Stopping container kube-rbac-proxy
17:00:50	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd	Killing	Stopping container kserve-container
17:00:53	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-btnjd	Unhealthy	Readiness probe failed: Get "https://10.133.0.51:8643/healthz": dial tcp 10.133.0.51:8643: connect: connection refused
17:00:56	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f	Created	Created container: storage-initializer
17:00:56	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f	AddedInterface	Add eth0 [10.132.0.47/23] from ovn-kubernetes
17:00:56	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5	SuccessfulCreate	Created pod: isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f
17:00:56	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f	Started	Started container storage-initializer
17:00:56	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-runtime-predictor-7f4957c5d5 from 0 to 1
17:00:56	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
17:00:56	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
17:01:01	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f	Started	Started container kserve-container
17:01:01	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
17:01:01	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f	Started	Started container kube-rbac-proxy
17:01:01	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f	Created	Created container: kube-rbac-proxy
17:01:01	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:01:01	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f	Created	Created container: kserve-container
17:01:02 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:01:02 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:01:26	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:01:26	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:01:39 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:01:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-runtime] is Ready
17:01:41 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:01:41 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:01:46	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f	Killing	Stopping container kserve-container
17:01:46	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f	Killing	Stopping container kube-rbac-proxy
17:01:51	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5 from 0 to 1
17:01:51	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
17:01:51	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
17:01:51	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5	SuccessfulCreate	Created pod: isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68
17:01:52	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68	Started	Started container storage-initializer
17:01:52	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68	AddedInterface	Add eth0 [10.133.0.53/23] from ovn-kubernetes
17:01:52	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
17:01:52	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68	Created	Created container: storage-initializer
17:01:53 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:01:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	InferenceServiceReady	InferenceService [isvc-sklearn-s3] is Ready
17:01:56	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68	Created	Created container: kube-rbac-proxy
17:01:56	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68	Started	Started container kube-rbac-proxy
17:01:56	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68	Pulled	Container image "kserve/lgbserver:latest" already present on machine
17:01:56	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68	Created	Created container: kserve-container
17:01:56	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68	Started	Started container kserve-container
17:01:56	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:02:02	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45 from 0 to 1
17:02:02	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-pass": the object has been modified; please apply your changes to the latest version and try again
17:02:02	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp
17:02:02 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:02:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-84d46f45-8fxg6	Killing	Stopping container kube-rbac-proxy
17:02:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-84d46f45-8fxg6	Killing	Stopping container kserve-container
17:02:02 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:02:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-84d46f45-8fxg6	Unhealthy	Readiness probe failed: Get "https://10.133.0.52:8643/healthz": dial tcp 10.133.0.52:8643: connect: connection refused
17:02:03 (x3)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68	Unhealthy	Readiness probe failed: dial tcp 10.133.0.53:8080: connect: connection refused
17:02:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp	Created	Created container: storage-initializer
17:02:03 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-84d46f45-8fxg6	Unhealthy	Readiness probe failed: dial tcp 10.133.0.52:8080: connect: connection refused
17:02:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
17:02:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp	Started	Started container storage-initializer
17:02:03	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp	AddedInterface	Add eth0 [10.132.0.48/23] from ovn-kubernetes
17:02:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp	Started	Started container kube-rbac-proxy
17:02:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp	Created	Created container: kube-rbac-proxy
17:02:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:02:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp	Started	Started container kserve-container
17:02:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp	Created	Created container: kserve-container
17:02:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine
17:02:06	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:02:06	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:02:13	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-kserve] is Ready
17:02:13 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:02:21	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68	Killing	Stopping container kserve-container
17:02:21	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:02:21	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedUpdateStatus	horizontalpodautoscalers.autoscaling "isvc-lightgbm-v2-kserve-predictor" not found
17:02:21	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:02:21	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ncg68	Killing	Stopping container kube-rbac-proxy
17:02:26	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-mlflow-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-mlflow-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
17:02:26	kserve-ci-e2e-test	deployment-controller	isvc-mlflow-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-mlflow-v2-runtime-predictor-6456f94494 from 0 to 1
17:02:26	kserve-ci-e2e-test	replicaset-controller	isvc-mlflow-v2-runtime-predictor-6456f94494	SuccessfulCreate	Created pod: isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8
17:02:27	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8	Created	Created container: storage-initializer
17:02:27	kserve-ci-e2e-test	multus	isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8	AddedInterface	Add eth0 [10.132.0.49/23] from ovn-kubernetes
17:02:27	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
17:02:27	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8	Started	Started container storage-initializer
17:02:31	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:02:31	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
17:02:31	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8	Created	Created container: kserve-container
17:02:31	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8	Started	Started container kube-rbac-proxy
17:02:31	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8	Created	Created container: kube-rbac-proxy
17:02:31	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8	Started	Started container kserve-container
17:02:32 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:02:32 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:03:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	InferenceServiceReady	InferenceService [isvc-mlflow-v2-runtime] is Ready
17:03:08 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:03:11 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:03:11 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:03:11 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp	Unhealthy	Readiness probe failed: dial tcp 10.132.0.48:8080: connect: connection refused
17:03:16	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8	Killing	Stopping container kube-rbac-proxy
17:03:16	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8	Killing	Stopping container kserve-container
17:03:17 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:03:17 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:03:21	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-global-pass] is Ready
17:03:21 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:03:21	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-mcp-predictor-8584bb4bfb	SuccessfulCreate	Created pod: isvc-sklearn-mcp-predictor-8584bb4bfb-nh268
17:03:21	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-mcp-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-mcp-predictor-8584bb4bfb from 0 to 1
17:03:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-8584bb4bfb-nh268	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
17:03:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp	Killing	Stopping container kserve-container
17:03:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-8584bb4bfb-nh268	Started	Started container storage-initializer
17:03:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-8584bb4bfb-nh268	Created	Created container: storage-initializer
17:03:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp	Killing	Stopping container kube-rbac-proxy
17:03:22 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-mcp": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-mcp": the object has been modified; please apply your changes to the latest version and try again
17:03:22	kserve-ci-e2e-test	multus	isvc-sklearn-mcp-predictor-8584bb4bfb-nh268	AddedInterface	Add eth0 [10.133.0.54/23] from ovn-kubernetes
17:03:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-mcp": the object has been modified; please apply your changes to the latest version and try again
17:03:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
17:03:23	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw
17:03:23	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669 from 0 to 1
17:03:23	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-fail": the object has been modified; please apply your changes to the latest version and try again
17:03:24	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw	AddedInterface	Add eth0 [10.133.0.55/23] from ovn-kubernetes
17:03:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp	Unhealthy	Readiness probe failed: Get "https://10.132.0.48:8643/healthz": dial tcp 10.132.0.48:8643: connect: connection refused
17:03:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-8584bb4bfb-nh268	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine
17:03:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-8584bb4bfb-nh268	Created	Created container: kserve-container
17:03:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-8584bb4bfb-nh268	Started	Started container kserve-container
17:03:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-8584bb4bfb-nh268	Pulling	Pulling image "quay.io/opendatahub/kserve-agent:latest"
17:03:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-8584bb4bfb-nh268	Created	Created container: kserve-agent
17:03:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-8584bb4bfb-nh268	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent:latest" in 2.882s (2.882s including waiting). Image size: 239481897 bytes.
17:03:29 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
17:03:29 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw	Created	Created container: storage-initializer
17:03:29 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw	Started	Started container storage-initializer
17:03:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-8584bb4bfb-nh268	Started	Started container kserve-agent
17:03:29 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:03:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-8584bb4bfb-nh268	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:03:30	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-8584bb4bfb-nh268	Started	Started container kube-rbac-proxy
17:03:30	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-8584bb4bfb-nh268	Created	Created container: kube-rbac-proxy
17:03:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-kq8lw	Killing	Stopping container storage-initializer
17:03:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-pass": the object has been modified; please apply your changes to the latest version and try again
17:03:34	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7 from 0 to 1
17:03:34	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f
17:03:35	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f	AddedInterface	Add eth0 [10.132.0.50/23] from ovn-kubernetes
17:03:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f	Started	Started container storage-initializer
17:03:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f	Created	Created container: storage-initializer
17:03:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
17:03:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:03:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f	Created	Created container: kube-rbac-proxy
17:03:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f	Started	Started container kserve-container
17:03:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f	Created	Created container: kserve-container
17:03:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine
17:03:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f	Started	Started container kube-rbac-proxy
17:03:49	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:03:49	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:04:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-8584bb4bfb-nh268	Unhealthy	Liveness probe failed: Get "http://10.133.0.54:8080/v1/models/isvc-sklearn-mcp": dial tcp 10.133.0.54:8080: connect: connection refused
17:04:37 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:04:37	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	InferenceServiceReady	InferenceService [isvc-sklearn-mcp] is Ready
17:04:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-8584bb4bfb-nh268	Killing	Stopping container kserve-agent
17:04:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-8584bb4bfb-nh268	Killing	Stopping container kube-rbac-proxy
17:04:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-8584bb4bfb-nh268	Killing	Stopping container kserve-container
17:04:47	kserve-ci-e2e-test	deployment-controller	isvc-paddle-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-predictor-6d5c9fd4f6 from 0 to 1
17:04:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	UpdateFailed	Failed to update status for InferenceService "isvc-paddle": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle": the object has been modified; please apply your changes to the latest version and try again
17:04:47	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-predictor-6d5c9fd4f6	SuccessfulCreate	Created pod: isvc-paddle-predictor-6d5c9fd4f6-xf4dn
17:04:47	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-xf4dn	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
17:04:47	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-xf4dn	Created	Created container: storage-initializer
17:04:47	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-xf4dn	Started	Started container storage-initializer
17:04:47	kserve-ci-e2e-test	multus	isvc-paddle-predictor-6d5c9fd4f6-xf4dn	AddedInterface	Add eth0 [10.133.0.56/23] from ovn-kubernetes
17:04:50 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:04:50 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:04:53	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-xf4dn	Pulling	Pulling image "kserve/paddleserver:latest"
17:04:55 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:04:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-custom-pass] is Ready
17:05:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f	Killing	Stopping container kserve-container
17:05:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f	Killing	Stopping container kube-rbac-proxy
17:05:05	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-xf4dn	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:05:05	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-xf4dn	Started	Started container kserve-container
17:05:05 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f	Unhealthy	Readiness probe failed: dial tcp 10.132.0.50:8080: connect: connection refused
17:05:05	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-xf4dn	Created	Created container: kserve-container
17:05:05	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-xf4dn	Pulled	Successfully pulled image "kserve/paddleserver:latest" in 12.123s (12.123s including waiting). Image size: 1175679746 bytes.
17:05:05	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-xf4dn	Created	Created container: kube-rbac-proxy
17:05:05	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-xf4dn	Started	Started container kube-rbac-proxy
17:05:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f	Unhealthy	Readiness probe failed: Get "https://10.132.0.50:8643/healthz": dial tcp 10.132.0.50:8643: connect: connection refused
17:05:06	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf
17:05:06 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-fail": the object has been modified; please apply your changes to the latest version and try again
17:05:06	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-fail": the object has been modified; please apply your changes to the latest version and try again
17:05:06 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-8584bb4bfb-nh268	Unhealthy	Readiness probe failed: Get "http://10.133.0.54:8080/v1/models/isvc-sklearn-mcp": dial tcp 10.133.0.54:8080: connect: connection refused
17:05:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
17:05:06	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f from 0 to 1
17:05:07	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf	AddedInterface	Add eth0 [10.132.0.51/23] from ovn-kubernetes
17:05:11 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:05:11 (x6)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-8584bb4bfb-nh268	Unhealthy	Readiness probe failed: Get "https://10.133.0.54:8643/healthz": dial tcp 10.133.0.54:8643: connect: connection refused
17:05:11 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
17:05:11 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf	Created	Created container: storage-initializer
17:05:11 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf	Started	Started container storage-initializer
17:05:16	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf	Killing	Stopping container storage-initializer
17:05:17	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8
17:05:17	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694 from 0 to 1
17:05:17	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-pass": the object has been modified; please apply your changes to the latest version and try again
17:05:17	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8	AddedInterface	Add eth0 [10.133.0.57/23] from ovn-kubernetes
17:05:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
17:05:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8	Created	Created container: storage-initializer
17:05:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8	Started	Started container storage-initializer
17:05:17	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-pass": the object has been modified; please apply your changes to the latest version and try again
17:05:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8	Started	Started container kserve-container
17:05:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8	Created	Created container: kube-rbac-proxy
17:05:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8	Created	Created container: kserve-container
17:05:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine
17:05:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:05:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8	Started	Started container kube-rbac-proxy
17:05:32 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:05:32 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:05:43 (x6)	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-xf4dn	Unhealthy	Readiness probe failed: dial tcp 10.133.0.56:8080: connect: connection refused
17:05:47 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:05:47	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:05:47 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:05:47	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:05:53 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:05:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	InferenceServiceReady	InferenceService [isvc-paddle] is Ready
17:05:58	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-xf4dn	Killing	Stopping container kserve-container
17:05:58	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-xf4dn	Unhealthy	Readiness probe failed: Get "https://10.133.0.56:8643/healthz": dial tcp 10.133.0.56:8643: connect: connection refused
17:05:58	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-xf4dn	Killing	Stopping container kube-rbac-proxy
17:06:03	kserve-ci-e2e-test	deployment-controller	isvc-paddle-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-runtime-predictor-58b99fff76 from 0 to 1
17:06:03	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-runtime-predictor-58b99fff76	SuccessfulCreate	Created pod: isvc-paddle-runtime-predictor-58b99fff76-njqm2
17:06:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-runtime": the object has been modified; please apply your changes to the latest version and try again
17:06:04	kserve-ci-e2e-test	multus	isvc-paddle-runtime-predictor-58b99fff76-njqm2	AddedInterface	Add eth0 [10.133.0.58/23] from ovn-kubernetes
17:06:04	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-njqm2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
17:06:04	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-njqm2	Created	Created container: storage-initializer
17:06:04	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-njqm2	Started	Started container storage-initializer
17:06:09	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-njqm2	Started	Started container kube-rbac-proxy
17:06:09	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-njqm2	Pulled	Container image "kserve/paddleserver:latest" already present on machine
17:06:09	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-njqm2	Created	Created container: kserve-container
17:06:09	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-njqm2	Started	Started container kserve-container
17:06:09	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-njqm2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:06:09	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-njqm2	Created	Created container: kube-rbac-proxy
17:06:32 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:06:32 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:06:33 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:06:33 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:06:41 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:06:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-serving-pass] is Ready
17:06:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8	Killing	Stopping container kserve-container
17:06:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8	Killing	Stopping container kube-rbac-proxy
17:06:48 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-fail": the object has been modified; please apply your changes to the latest version and try again
17:06:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
17:06:48	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-fail": the object has been modified; please apply your changes to the latest version and try again
17:06:48	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d from 0 to 1
17:06:48	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl
17:06:49	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl	AddedInterface	Add eth0 [10.132.0.52/23] from ovn-kubernetes
17:06:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8	Unhealthy	Readiness probe failed: Get "https://10.133.0.57:8643/healthz": dial tcp 10.133.0.57:8643: connect: connection refused
17:06:51 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-8zkl8	Unhealthy	Readiness probe failed: dial tcp 10.133.0.57:8080: connect: connection refused
17:06:53 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
17:06:54 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl	Created	Created container: storage-initializer
17:06:54 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl	Started	Started container storage-initializer
17:06:54 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:06:57	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	InferenceServiceReady	InferenceService [isvc-paddle-runtime] is Ready
17:06:57 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:06:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl	Killing	Stopping container storage-initializer
17:06:59	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-runtime-predictor-885fc8777	SuccessfulCreate	Created pod: isvc-pmml-runtime-predictor-885fc8777-lfvrb
17:06:59	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-runtime": the object has been modified; please apply your changes to the latest version and try again
17:06:59	kserve-ci-e2e-test	deployment-controller	isvc-pmml-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-runtime-predictor-885fc8777 from 0 to 1
17:07:00	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-lfvrb	Created	Created container: storage-initializer
17:07:00	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-lfvrb	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
17:07:00	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-lfvrb	Started	Started container storage-initializer
17:07:00	kserve-ci-e2e-test	multus	isvc-pmml-runtime-predictor-885fc8777-lfvrb	AddedInterface	Add eth0 [10.133.0.59/23] from ovn-kubernetes
17:07:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:07:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:07:04	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-lfvrb	Pulling	Pulling image "kserve/pmmlserver:latest"
17:07:04	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-njqm2	Killing	Stopping container kube-rbac-proxy
17:07:04	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-njqm2	Killing	Stopping container kserve-container
17:07:07 (x7)	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-njqm2	Unhealthy	Readiness probe failed: dial tcp 10.133.0.58:8080: connect: connection refused
17:07:07	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-njqm2	Unhealthy	Readiness probe failed: Get "https://10.133.0.58:8643/healthz": dial tcp 10.133.0.58:8643: connect: connection refused
17:07:09	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-v2-kserve-predictor-64d4c74b4d	SuccessfulCreate	Created pod: isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9
17:07:09	kserve-ci-e2e-test	deployment-controller	isvc-paddle-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-v2-kserve-predictor-64d4c74b4d from 0 to 1
17:07:09	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
17:07:12	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-lfvrb	Started	Started container kube-rbac-proxy
17:07:12	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9	Created	Created container: storage-initializer
17:07:12	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-lfvrb	Created	Created container: kube-rbac-proxy
17:07:12	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-lfvrb	Pulled	Successfully pulled image "kserve/pmmlserver:latest" in 7.346s (7.346s including waiting). Image size: 826851186 bytes.
17:07:12	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-lfvrb	Created	Created container: kserve-container
17:07:12	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-lfvrb	Started	Started container kserve-container
17:07:12	kserve-ci-e2e-test	multus	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9	AddedInterface	Add eth0 [10.133.0.60/23] from ovn-kubernetes
17:07:12	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
17:07:12	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-lfvrb	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:07:12	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9	Started	Started container storage-initializer
17:07:17	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9	Pulled	Container image "kserve/paddleserver:latest" already present on machine
17:07:18	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9	Created	Created container: kserve-container
17:07:18	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9	Started	Started container kserve-container
17:07:18	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:07:18	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9	Created	Created container: kube-rbac-proxy
17:07:18	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9	Started	Started container kube-rbac-proxy
17:07:44 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:07:44 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:07:54 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:07:54 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:07:55 (x6)	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9	Unhealthy	Readiness probe failed: dial tcp 10.133.0.60:8080: connect: connection refused
17:08:06 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:08:06	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	InferenceServiceReady	InferenceService [isvc-paddle-v2-kserve] is Ready
17:08:09	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:08:09	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:08:11	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9	Killing	Stopping container kserve-container
17:08:11	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-wr5d9	Killing	Stopping container kube-rbac-proxy
17:08:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	UpdateFailed	Failed to update status for InferenceService "isvc-pmml": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml": the object has been modified; please apply your changes to the latest version and try again
17:08:16	kserve-ci-e2e-test	deployment-controller	isvc-pmml-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-predictor-8674795f59 from 0 to 1
17:08:16	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-predictor-8674795f59	SuccessfulCreate	Created pod: isvc-pmml-predictor-8674795f59-bnjbw
17:08:16	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-bnjbw	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-pmml-predictor-serving-cert" not found
17:08:17	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-bnjbw	Started	Started container storage-initializer
17:08:17	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-bnjbw	Created	Created container: storage-initializer
17:08:17	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-bnjbw	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
17:08:17	kserve-ci-e2e-test	multus	isvc-pmml-predictor-8674795f59-bnjbw	AddedInterface	Add eth0 [10.132.0.53/23] from ovn-kubernetes
17:08:22	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-bnjbw	Pulling	Pulling image "kserve/pmmlserver:latest"
17:08:29	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-bnjbw	Created	Created container: kube-rbac-proxy
17:08:29 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-lfvrb	Unhealthy	Readiness probe failed: dial tcp 10.133.0.59:8080: connect: connection refused
17:08:29	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-bnjbw	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:08:29	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-bnjbw	Started	Started container kserve-container
17:08:29	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-bnjbw	Created	Created container: kserve-container
17:08:29	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-bnjbw	Pulled	Successfully pulled image "kserve/pmmlserver:latest" in 7.092s (7.092s including waiting). Image size: 826851186 bytes.
17:08:29	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-bnjbw	Started	Started container kube-rbac-proxy
17:08:31	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:08:31	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:08:40 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:08:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	InferenceServiceReady	InferenceService [isvc-pmml-runtime] is Ready
17:08:44 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:08:44 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:08:50	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-lfvrb	Killing	Stopping container kube-rbac-proxy
17:08:50	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-lfvrb	Killing	Stopping container kserve-container
17:08:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
17:08:55	kserve-ci-e2e-test	deployment-controller	isvc-pmml-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-v2-kserve-predictor-84cb55486c from 0 to 1
17:08:55	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-v2-kserve-predictor-84cb55486c	SuccessfulCreate	Created pod: isvc-pmml-v2-kserve-predictor-84cb55486c-286wj
17:08:56	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-286wj	Started	Started container storage-initializer
17:08:56	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-286wj	Created	Created container: storage-initializer
17:08:56	kserve-ci-e2e-test	multus	isvc-pmml-v2-kserve-predictor-84cb55486c-286wj	AddedInterface	Add eth0 [10.133.0.61/23] from ovn-kubernetes
17:08:56	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-286wj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
17:09:00	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-286wj	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
17:09:00	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-286wj	Created	Created container: kserve-container
17:09:00	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-286wj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:09:00	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-286wj	Started	Started container kserve-container
17:09:00	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-286wj	Started	Started container kube-rbac-proxy
17:09:00	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-286wj	Created	Created container: kube-rbac-proxy
17:09:40 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:09:40 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:09:42 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-bnjbw	Unhealthy	Readiness probe failed: dial tcp 10.132.0.53:8080: connect: connection refused
17:09:46 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:09:46 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:09:52 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:09:52	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	InferenceServiceReady	InferenceService [isvc-pmml] is Ready
17:09:57	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-bnjbw	Killing	Stopping container kserve-container
17:09:57	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-bnjbw	Killing	Stopping container kube-rbac-proxy
17:10:17 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-286wj	Unhealthy	Readiness probe failed: dial tcp 10.133.0.61:8080: connect: connection refused
17:10:25 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:10:25 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:10:26	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	InferenceServiceReady	InferenceService [isvc-pmml-v2-kserve] is Ready
17:10:26 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:10:36	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-286wj	Killing	Stopping container kserve-container
17:10:36	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-286wj	Killing	Stopping container kube-rbac-proxy
17:10:37	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-286wj	Unhealthy	Readiness probe failed: Get "https://10.133.0.61:8643/healthz": dial tcp 10.133.0.61:8643: connect: connection refused
17:10:42	kserve-ci-e2e-test	kubelet	isvc-primary-932c75-predictor-bd6c947bf-bhkvw	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-primary-932c75-predictor-serving-cert" not found
17:10:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-932c75	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-932c75": the object has been modified; please apply your changes to the latest version and try again
17:10:42	kserve-ci-e2e-test	deployment-controller	isvc-primary-932c75-predictor	ScalingReplicaSet	Scaled up replica set isvc-primary-932c75-predictor-bd6c947bf from 0 to 1
17:10:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-932c75	UpdateFailed	Failed to update status for InferenceService "isvc-primary-932c75": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-932c75": the object has been modified; please apply your changes to the latest version and try again
17:10:42	kserve-ci-e2e-test	replicaset-controller	isvc-primary-932c75-predictor-bd6c947bf	SuccessfulCreate	Created pod: isvc-primary-932c75-predictor-bd6c947bf-bhkvw
17:10:43	kserve-ci-e2e-test	kubelet	isvc-primary-932c75-predictor-bd6c947bf-bhkvw	Created	Created container: storage-initializer
17:10:43	kserve-ci-e2e-test	kubelet	isvc-primary-932c75-predictor-bd6c947bf-bhkvw	Started	Started container storage-initializer
17:10:43	kserve-ci-e2e-test	kubelet	isvc-primary-932c75-predictor-bd6c947bf-bhkvw	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
17:10:43	kserve-ci-e2e-test	multus	isvc-primary-932c75-predictor-bd6c947bf-bhkvw	AddedInterface	Add eth0 [10.133.0.62/23] from ovn-kubernetes
17:10:47	kserve-ci-e2e-test	kubelet	isvc-primary-932c75-predictor-bd6c947bf-bhkvw	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1928" already present on machine
17:10:47	kserve-ci-e2e-test	kubelet	isvc-primary-932c75-predictor-bd6c947bf-bhkvw	Created	Created container: kserve-container
17:10:47	kserve-ci-e2e-test	kubelet	isvc-primary-932c75-predictor-bd6c947bf-bhkvw	Started	Started container kserve-container
17:10:47	kserve-ci-e2e-test	kubelet	isvc-primary-932c75-predictor-bd6c947bf-bhkvw	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:10:47	kserve-ci-e2e-test	kubelet	isvc-primary-932c75-predictor-bd6c947bf-bhkvw	Created	Created container: kube-rbac-proxy
17:10:47	kserve-ci-e2e-test	kubelet	isvc-primary-932c75-predictor-bd6c947bf-bhkvw	Started	Started container kube-rbac-proxy
17:11:27 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-932c75-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:11:27 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-932c75-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:12:05	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-932c75	InferenceServiceReady	InferenceService [isvc-primary-932c75] is Ready
17:12:06 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-932c75	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:12:27 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-932c75	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-932c75": the object has been modified; please apply your changes to the latest version and try again
17:12:27 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-932c75	UpdateFailed	Failed to update status for InferenceService "isvc-secondary-932c75": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-932c75": the object has been modified; please apply your changes to the latest version and try again
17:12:27	kserve-ci-e2e-test	deployment-controller	isvc-secondary-932c75-predictor	ScalingReplicaSet	Scaled up replica set isvc-secondary-932c75-predictor-76c5bfcd from 0 to 1
17:12:27	kserve-ci-e2e-test	replicaset-controller	isvc-secondary-932c75-predictor-76c5bfcd	SuccessfulCreate	Created pod: isvc-secondary-932c75-predictor-76c5bfcd-xvcc9
17:12:27	kserve-ci-e2e-test	kubelet	isvc-secondary-932c75-predictor-76c5bfcd-xvcc9	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-secondary-932c75-predictor-serving-cert" not found
17:12:28	kserve-ci-e2e-test	multus	isvc-secondary-932c75-predictor-76c5bfcd-xvcc9	AddedInterface	Add eth0 [10.133.0.63/23] from ovn-kubernetes
17:12:33 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-932c75-predictor-76c5bfcd-xvcc9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4164f82a611b9e52a903d9686e415dbc9c35d0c556e392d7254ca59528ec9948" already present on machine
17:12:33 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-932c75-predictor-76c5bfcd-xvcc9	Created	Created container: storage-initializer
17:12:33 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-932c75-predictor-76c5bfcd-xvcc9	Started	Started container storage-initializer
17:12:39	kserve-ci-e2e-test	kubelet	isvc-secondary-932c75-predictor-76c5bfcd-xvcc9	BackOff	Back-off restarting failed container storage-initializer in pod isvc-secondary-932c75-predictor-76c5bfcd-xvcc9_kserve-ci-e2e-test(9999eec3-90ad-49a1-85fa-c21c230c2653)
17:12:39 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-932c75	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:12:42 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-932c75-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:12:42 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-932c75-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:12:42	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-932c75-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:12:42	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-932c75-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:12:45 (x10)	kserve-ci-e2e-test	kubelet	isvc-primary-932c75-predictor-bd6c947bf-bhkvw	Unhealthy	Readiness probe failed: dial tcp 10.133.0.62:8080: connect: connection refused
17:12:45	kserve-ci-e2e-test	kubelet	isvc-primary-932c75-predictor-bd6c947bf-bhkvw	Killing	Stopping container kserve-container
17:12:45	kserve-ci-e2e-test	kubelet	isvc-primary-932c75-predictor-bd6c947bf-bhkvw	Unhealthy	Readiness probe failed: Get "https://10.133.0.62:8643/healthz": dial tcp 10.133.0.62:8643: connect: connection refused
17:12:45	kserve-ci-e2e-test	kubelet	isvc-primary-932c75-predictor-bd6c947bf-bhkvw	Killing	Stopping container kube-rbac-proxy