Events

Time	Namespace	Component	RelatedObject	Reason	Message
22:20:47	kserve-ci-e2e-test		isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq to ip-10-0-143-243.ec2.internal
22:45:29	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv to ip-10-0-143-243.ec2.internal
22:42:15	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8 to ip-10-0-129-163.ec2.internal
21:44:04	kserve-ci-e2e-test		isvc-canary-promote-predictor-74874cc84b-7b45l	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l to ip-10-0-129-163.ec2.internal
21:45:34	kserve-ci-e2e-test		isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd to ip-10-0-143-243.ec2.internal
21:44:04	kserve-ci-e2e-test		isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq to ip-10-0-129-163.ec2.internal
21:48:59	kserve-ci-e2e-test		isvc-canary-rollback-predictor-6d775fdd5d-l5l4f	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f to ip-10-0-129-163.ec2.internal
21:47:29	kserve-ci-e2e-test		isvc-canary-rollback-predictor-6d775fdd5d-ll64b	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b to ip-10-0-143-243.ec2.internal
21:47:29	kserve-ci-e2e-test		isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2 to ip-10-0-143-243.ec2.internal
21:50:35	kserve-ci-e2e-test		isvc-canary-stop-predictor-8bd7d86fd-jwvtx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx to ip-10-0-143-243.ec2.internal
21:50:35	kserve-ci-e2e-test		isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd to ip-10-0-143-243.ec2.internal
22:11:55	kserve-ci-e2e-test		isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26 to ip-10-0-143-243.ec2.internal
22:47:00	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q to ip-10-0-129-163.ec2.internal
21:54:00	kserve-ci-e2e-test		isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm to ip-10-0-143-243.ec2.internal
21:58:32	kserve-ci-e2e-test		isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968 to ip-10-0-143-243.ec2.internal
21:55:36	kserve-ci-e2e-test		isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn to ip-10-0-143-243.ec2.internal
21:29:58	kserve-ci-e2e-test		isvc-logger-predictor-5f586b99d8-ckct4	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4 to ip-10-0-129-163.ec2.internal
21:42:29	kserve-ci-e2e-test		isvc-canary-create-predictor-7f74cfd478-88wlx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx to ip-10-0-129-163.ec2.internal
21:59:07	kserve-ci-e2e-test		isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl to ip-10-0-143-243.ec2.internal
22:01:27	kserve-ci-e2e-test		isvc-paddle-predictor-6d5c9fd4f6-fcrw2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2 to ip-10-0-129-163.ec2.internal
22:02:44	kserve-ci-e2e-test		isvc-paddle-runtime-predictor-58b99fff76-v9vk8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8 to ip-10-0-143-243.ec2.internal
22:04:01	kserve-ci-e2e-test		isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2 to ip-10-0-129-163.ec2.internal
22:05:08	kserve-ci-e2e-test		isvc-pmml-predictor-8674795f59-cvmzv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv to ip-10-0-143-243.ec2.internal
22:06:44	kserve-ci-e2e-test		isvc-pmml-runtime-predictor-885fc8777-nmh2s	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s to ip-10-0-143-243.ec2.internal
21:41:01	kserve-ci-e2e-test		isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh to ip-10-0-129-163.ec2.internal
22:08:10	kserve-ci-e2e-test		isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt to ip-10-0-143-243.ec2.internal
22:00:02	kserve-ci-e2e-test		isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv to ip-10-0-143-243.ec2.internal
21:42:29	kserve-ci-e2e-test		isvc-canary-create-v2-predictor-75c7994655-s4p7r	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r to ip-10-0-129-163.ec2.internal
22:36:23	kserve-ci-e2e-test		xgboost-v2-mlserver-predictor-59944bcf4c-pztsc	Scheduled	Successfully assigned kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc to ip-10-0-143-243.ec2.internal
22:24:05	kserve-ci-e2e-test		sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4	Scheduled	Successfully assigned kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4 to ip-10-0-143-243.ec2.internal
21:36:37	kserve-ci-e2e-test		isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4 to ip-10-0-129-163.ec2.internal
22:12:11	kserve-ci-e2e-test		isvc-predictive-sklearn-predictor-5f849d69df-z4q42	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42 to ip-10-0-143-243.ec2.internal
22:17:07	kserve-ci-e2e-test		isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg to ip-10-0-129-163.ec2.internal
21:34:25	kserve-ci-e2e-test		isvc-autogluon-v2-predictor-9684b4cd9-9g7nc	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc to ip-10-0-129-163.ec2.internal
21:29:38	kserve-ci-e2e-test		message-dumper-predictor-6575bcfdfd-pfc7l	Scheduled	Successfully assigned kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l to ip-10-0-129-163.ec2.internal
21:26:17	kserve-ci-e2e-test		isvc-sklearn-batcher-predictor-7776d964d7-w2t6x	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x to ip-10-0-129-163.ec2.internal
22:38:34	kserve-ci-e2e-test		isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp to ip-10-0-143-243.ec2.internal
21:41:48	kserve-ci-e2e-test		isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2 to ip-10-0-129-163.ec2.internal
22:39:39	kserve-ci-e2e-test		isvc-xgboost-v2-predictor-fcc668bc-m76lz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz to ip-10-0-143-243.ec2.internal
22:35:28	kserve-ci-e2e-test		isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh to ip-10-0-143-243.ec2.internal
22:37:18	kserve-ci-e2e-test		isvc-xgboost-runtime-predictor-54dd488c85-bwgjp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp to ip-10-0-143-243.ec2.internal
22:33:53	kserve-ci-e2e-test		isvc-xgboost-predictor-7566d65bf7-tc894	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894 to ip-10-0-143-243.ec2.internal
22:31:35	kserve-ci-e2e-test		isvc-triton-predictor-5d7db58676-cztrf	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf to ip-10-0-129-163.ec2.internal
22:30:50	kserve-ci-e2e-test		isvc-tensorflow-runtime-predictor-855cfcd677-7699m	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m to ip-10-0-143-243.ec2.internal
22:30:03	kserve-ci-e2e-test		isvc-tensorflow-predictor-547dfc5669-hnppd	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd to ip-10-0-129-163.ec2.internal
22:25:47	kserve-ci-e2e-test		isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6 to ip-10-0-143-243.ec2.internal
22:26:53	kserve-ci-e2e-test		isvc-sklearn-v2-predictor-59d6dcff47-8nhk6	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6 to ip-10-0-143-243.ec2.internal
21:31:23	kserve-ci-e2e-test		isvc-autogluon-v1-predictor-574b5b7495-76zsh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh to ip-10-0-129-163.ec2.internal
22:28:28	kserve-ci-e2e-test		isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d to ip-10-0-143-243.ec2.internal
22:15:31	kserve-ci-e2e-test		isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz to ip-10-0-143-243.ec2.internal
21:52:15	kserve-ci-e2e-test		isvc-lightgbm-predictor-54f85ddf88-kndbs	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs to ip-10-0-143-243.ec2.internal
22:43:46	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t to ip-10-0-143-243.ec2.internal
22:14:06	kserve-ci-e2e-test		isvc-predictive-xgboost-predictor-695f978978-64tnq	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq to ip-10-0-143-243.ec2.internal
22:19:12	kserve-ci-e2e-test		isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c to ip-10-0-143-243.ec2.internal
22:43:57	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8 to ip-10-0-143-243.ec2.internal
22:45:18	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj to ip-10-0-129-163.ec2.internal
22:40:54	kserve-ci-e2e-test		isvc-sklearn-s3-predictor-7d6947b598-66vn6	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6 to ip-10-0-143-243.ec2.internal
21:38:39	kserve-ci-e2e-test		isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq to ip-10-0-129-163.ec2.internal
22:25:10	kserve-ci-e2e-test		isvc-sklearn-runtime-predictor-cf66df4c5-64kkx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx to ip-10-0-129-163.ec2.internal
22:22:30	kserve-ci-e2e-test		isvc-sklearn-predictor-6c799687d7-2qq4t	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t to ip-10-0-143-243.ec2.internal
22:09:47	kserve-ci-e2e-test		isvc-primary-5c6eb3-predictor-5587458446-fllrp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp to ip-10-0-143-243.ec2.internal
22:11:32	kserve-ci-e2e-test		isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz to ip-10-0-143-243.ec2.internal
21:28:03	kserve-ci-e2e-test		isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97 to ip-10-0-129-163.ec2.internal
21:26:17	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
21:26:17	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-predictor-7776d964d7	SuccessfulCreate	Created pod: isvc-sklearn-batcher-predictor-7776d964d7-w2t6x
21:26:17	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-predictor-7776d964d7 from 0 to 1
21:26:17	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
21:26:18	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-predictor-7776d964d7-w2t6x	AddedInterface	Add eth0 [10.134.0.29/23] from ovn-kubernetes
21:26:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7776d964d7-w2t6x	Started	Started container storage-initializer
21:26:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7776d964d7-w2t6x	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
21:26:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7776d964d7-w2t6x	Created	Created container: storage-initializer
21:26:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7776d964d7-w2t6x	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716"
21:26:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7776d964d7-w2t6x	Created	Created container: kserve-container
21:26:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7776d964d7-w2t6x	Started	Started container kserve-container
21:26:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7776d964d7-w2t6x	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" in 12.816s (12.816s including waiting). Image size: 1557885864 bytes.
21:26:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7776d964d7-w2t6x	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
21:26:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7776d964d7-w2t6x	Pulling	Pulling image "quay.io/opendatahub/kserve-agent@sha256:5eadda1fc9085300923462d30bccc33b4ca27b6d1bc72292ef033cbc042de960"
21:26:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7776d964d7-w2t6x	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.17s (2.17s including waiting). Image size: 211946088 bytes.
21:26:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7776d964d7-w2t6x	Started	Started container kube-rbac-proxy
21:26:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7776d964d7-w2t6x	Created	Created container: kube-rbac-proxy
21:26:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7776d964d7-w2t6x	Created	Created container: agent
21:26:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7776d964d7-w2t6x	Started	Started container agent
21:26:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7776d964d7-w2t6x	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:5eadda1fc9085300923462d30bccc33b4ca27b6d1bc72292ef033cbc042de960" in 2.12s (2.12s including waiting). Image size: 240177254 bytes.
21:27:17 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:27:17 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:27:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InferenceServiceReady	InferenceService [isvc-sklearn-batcher] is Ready
21:27:55 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:28:03 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:28:03 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:28:03	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-custom-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-custom-predictor-694d6c969d from 0 to 1
21:28:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7776d964d7-w2t6x	Killing	Stopping container kube-rbac-proxy
21:28:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher-custom": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher-custom": the object has been modified; please apply your changes to the latest version and try again
21:28:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7776d964d7-w2t6x	Killing	Stopping container kserve-container
21:28:03	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-custom-predictor-694d6c969d	SuccessfulCreate	Created pod: isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97
21:28:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
21:28:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7776d964d7-w2t6x	Killing	Stopping container agent
21:28:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97	Started	Started container storage-initializer
21:28:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97	Created	Created container: storage-initializer
21:28:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
21:28:04	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97	AddedInterface	Add eth0 [10.134.0.30/23] from ovn-kubernetes
21:28:05 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7776d964d7-w2t6x	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
21:28:05 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7776d964d7-w2t6x	Unhealthy	Readiness probe failed: dial tcp 10.134.0.29:8080: connect: connection refused
21:28:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97	Created	Created container: agent
21:28:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97	Started	Started container kube-rbac-proxy
21:28:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine
21:28:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97	Created	Created container: kserve-container
21:28:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97	Started	Started container kserve-container
21:28:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:28:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97	Created	Created container: kube-rbac-proxy
21:28:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97	Started	Started container agent
21:28:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:5eadda1fc9085300923462d30bccc33b4ca27b6d1bc72292ef033cbc042de960" already present on machine
21:28:13 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7776d964d7-w2t6x	Unhealthy	Readiness probe failed: Get "https://10.134.0.29:8643/healthz": dial tcp 10.134.0.29:8643: connect: connection refused
21:28:48 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:28:48 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:29:26 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:29:26	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	InferenceServiceReady	InferenceService [isvc-sklearn-batcher-custom] is Ready
21:29:33 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:29:33 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:29:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97	Killing	Stopping container kube-rbac-proxy
21:29:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97	Killing	Stopping container agent
21:29:38	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper": the object has been modified; please apply your changes to the latest version and try again
21:29:38 (x2)	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	UpdateFailed	Failed to update status for InferenceService "message-dumper": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper": the object has been modified; please apply your changes to the latest version and try again
21:29:38	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-pfc7l	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "message-dumper-predictor-serving-cert" not found
21:29:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97	Killing	Stopping container kserve-container
21:29:38	kserve-ci-e2e-test	deployment-controller	message-dumper-predictor	ScalingReplicaSet	Scaled up replica set message-dumper-predictor-6575bcfdfd from 0 to 1
21:29:38	kserve-ci-e2e-test	replicaset-controller	message-dumper-predictor-6575bcfdfd	SuccessfulCreate	Created pod: message-dumper-predictor-6575bcfdfd-pfc7l
21:29:39	kserve-ci-e2e-test	multus	message-dumper-predictor-6575bcfdfd-pfc7l	AddedInterface	Add eth0 [10.134.0.31/23] from ovn-kubernetes
21:29:39	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-pfc7l	Pulling	Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display"
21:29:40	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-pfc7l	Started	Started container kube-rbac-proxy
21:29:40	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-pfc7l	Pulled	Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.283s (1.283s including waiting). Image size: 14813193 bytes.
21:29:40	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-pfc7l	Created	Created container: kserve-container
21:29:40	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-pfc7l	Started	Started container kserve-container
21:29:40	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-pfc7l	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:29:40	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-pfc7l	Created	Created container: kube-rbac-proxy
21:29:46 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
21:29:46 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97	Unhealthy	Readiness probe failed: dial tcp 10.134.0.30:5000: connect: connection refused
21:29:49	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	InferenceServiceReady	InferenceService [message-dumper] is Ready
21:29:49 (x9)	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:29:51 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97	Unhealthy	Readiness probe failed: Get "https://10.134.0.30:8643/healthz": dial tcp 10.134.0.30:8643: connect: connection refused
21:29:58	kserve-ci-e2e-test	replicaset-controller	isvc-logger-predictor-5f586b99d8	SuccessfulCreate	Created pod: isvc-logger-predictor-5f586b99d8-ckct4
21:29:58	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5f586b99d8-ckct4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
21:29:58	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5f586b99d8-ckct4	Created	Created container: storage-initializer
21:29:58	kserve-ci-e2e-test	multus	isvc-logger-predictor-5f586b99d8-ckct4	AddedInterface	Add eth0 [10.134.0.32/23] from ovn-kubernetes
21:29:58	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5f586b99d8-ckct4	Started	Started container storage-initializer
21:29:58	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	UpdateFailed	Failed to update status for InferenceService "isvc-logger": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger": the object has been modified; please apply your changes to the latest version and try again
21:29:58	kserve-ci-e2e-test	deployment-controller	isvc-logger-predictor	ScalingReplicaSet	Scaled up replica set isvc-logger-predictor-5f586b99d8 from 0 to 1
21:30:03	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5f586b99d8-ckct4	Created	Created container: kube-rbac-proxy
21:30:03	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5f586b99d8-ckct4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:30:03	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5f586b99d8-ckct4	Started	Started container kserve-container
21:30:03	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5f586b99d8-ckct4	Started	Started container kube-rbac-proxy
21:30:03	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5f586b99d8-ckct4	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:5eadda1fc9085300923462d30bccc33b4ca27b6d1bc72292ef033cbc042de960" already present on machine
21:30:03	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5f586b99d8-ckct4	Created	Created container: agent
21:30:03	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5f586b99d8-ckct4	Started	Started container agent
21:30:03	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5f586b99d8-ckct4	Created	Created container: kserve-container
21:30:03	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5f586b99d8-ckct4	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine
21:30:08	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:30:08	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:30:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:30:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:30:28 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:30:28 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:31:13 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:31:13 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:31:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	InferenceServiceReady	InferenceService [isvc-logger] is Ready
21:31:16 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:31:23	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5f586b99d8-ckct4	Killing	Stopping container kube-rbac-proxy
21:31:23	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v1-predictor-574b5b7495	SuccessfulCreate	Created pod: isvc-autogluon-v1-predictor-574b5b7495-76zsh
21:31:23	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-pfc7l	Killing	Stopping container kserve-container
21:31:23	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5f586b99d8-ckct4	Killing	Stopping container agent
21:31:23	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5f586b99d8-ckct4	Killing	Stopping container kserve-container
21:31:23	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-pfc7l	Killing	Stopping container kube-rbac-proxy
21:31:23	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-76zsh	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-autogluon-v1-predictor-serving-cert" not found
21:31:23	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v1-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v1-predictor-574b5b7495 from 0 to 1
21:31:23	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v1": the object has been modified; please apply your changes to the latest version and try again
21:31:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-76zsh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
21:31:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-76zsh	Created	Created container: storage-initializer
21:31:24	kserve-ci-e2e-test	multus	isvc-autogluon-v1-predictor-574b5b7495-76zsh	AddedInterface	Add eth0 [10.134.0.33/23] from ovn-kubernetes
21:31:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-76zsh	Started	Started container storage-initializer
21:31:31 (x2)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5f586b99d8-ckct4	Unhealthy	Readiness probe failed: Get "https://10.134.0.32:8643/healthz": dial tcp 10.134.0.32:8643: connect: connection refused
21:31:36 (x11)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5f586b99d8-ckct4	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
21:31:36 (x12)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5f586b99d8-ckct4	Unhealthy	Readiness probe failed: dial tcp 10.134.0.32:8080: connect: connection refused
21:31:45	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-76zsh	Pulling	Pulling image "kserve/autogluonserver:latest"
21:33:47	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-76zsh	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:33:47	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-76zsh	Pulled	Successfully pulled image "kserve/autogluonserver:latest" in 2m1.757s (2m1.757s including waiting). Image size: 8940333847 bytes.
21:33:47	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-76zsh	Created	Created container: kube-rbac-proxy
21:33:47	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-76zsh	Started	Started container kserve-container
21:33:47	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-76zsh	Created	Created container: kserve-container
21:33:48	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-76zsh	Started	Started container kube-rbac-proxy
21:33:55 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-76zsh	Unhealthy	Readiness probe failed: dial tcp 10.134.0.33:8080: connect: connection refused
21:34:05 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:34:05	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	InferenceServiceReady	InferenceService [isvc-autogluon-v1] is Ready
21:34:08 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:34:08 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:34:20	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-76zsh	Killing	Stopping container kube-rbac-proxy
21:34:20	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-76zsh	Unhealthy	Readiness probe failed: Get "https://10.134.0.33:8643/healthz": dial tcp 10.134.0.33:8643: connect: connection refused
21:34:20	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-574b5b7495-76zsh	Killing	Stopping container kserve-container
21:34:25	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-9g7nc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
21:34:25	kserve-ci-e2e-test	multus	isvc-autogluon-v2-predictor-9684b4cd9-9g7nc	AddedInterface	Add eth0 [10.134.0.34/23] from ovn-kubernetes
21:34:25	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-predictor-9684b4cd9 from 0 to 1
21:34:25	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2": the object has been modified; please apply your changes to the latest version and try again
21:34:25	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-predictor-9684b4cd9	SuccessfulCreate	Created pod: isvc-autogluon-v2-predictor-9684b4cd9-9g7nc
21:34:26	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-9g7nc	Started	Started container storage-initializer
21:34:26	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-9g7nc	Created	Created container: storage-initializer
21:34:46	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-9g7nc	Started	Started container kube-rbac-proxy
21:34:46	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-9g7nc	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
21:34:46	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-9g7nc	Created	Created container: kserve-container
21:34:46	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-9g7nc	Started	Started container kserve-container
21:34:46	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-9g7nc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:34:46	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-9g7nc	Created	Created container: kube-rbac-proxy
21:35:10 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:35:10 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:36:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	InferenceServiceReady	InferenceService [isvc-autogluon-v2] is Ready
21:36:16 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:36:25 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:36:25 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:36:27	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-9g7nc	Unhealthy	Readiness probe failed: Get "http://10.134.0.34:8080/v2/models/isvc-autogluon-v2/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
21:36:31	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-9g7nc	Killing	Stopping container kube-rbac-proxy
21:36:31	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-9684b4cd9-9g7nc	Killing	Stopping container kserve-container
21:36:37	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2-variants": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-variants": the object has been modified; please apply your changes to the latest version and try again
21:36:37	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-variants-predictor-5c889d8844	SuccessfulCreate	Created pod: isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4
21:36:37	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
21:36:37	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-variants-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-variants-predictor-5c889d8844 from 0 to 1
21:36:38	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4	Started	Started container storage-initializer
21:36:38	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4	Created	Created container: storage-initializer
21:36:38	kserve-ci-e2e-test	multus	isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4	AddedInterface	Add eth0 [10.134.0.35/23] from ovn-kubernetes
21:36:38	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
21:36:43	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4	Started	Started container kserve-container
21:36:43	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:36:43	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4	Created	Created container: kube-rbac-proxy
21:36:43	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4	Created	Created container: kserve-container
21:36:43	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
21:36:43	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4	Started	Started container kube-rbac-proxy
21:37:22 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:37:22 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:38:17	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	InferenceServiceReady	InferenceService [isvc-autogluon-v2-variants] is Ready
21:38:17 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:38:22 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:38:22 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:38:28	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4	Unhealthy	Readiness probe failed: Get "http://10.134.0.35:8080/v2/models/isvc-autogluon-v2-variants/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
21:38:34	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4	Killing	Stopping container kube-rbac-proxy
21:38:34	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4	Killing	Stopping container kserve-container
21:38:39	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
21:38:39	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq	Started	Started container storage-initializer
21:38:39	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq	Created	Created container: storage-initializer
21:38:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2-noslash": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-noslash": the object has been modified; please apply your changes to the latest version and try again
21:38:39	kserve-ci-e2e-test	multus	isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq	AddedInterface	Add eth0 [10.134.0.36/23] from ovn-kubernetes
21:38:39	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-noslash-predictor-5499c79f9f	SuccessfulCreate	Created pod: isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq
21:38:39	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-noslash-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-noslash-predictor-5499c79f9f from 0 to 1
21:39:10	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
21:39:10	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq	Started	Started container kube-rbac-proxy
21:39:10	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq	Created	Created container: kserve-container
21:39:10	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq	Started	Started container kserve-container
21:39:10	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:39:10	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq	Created	Created container: kube-rbac-proxy
21:39:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:39:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:40:41 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:40:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	InferenceServiceReady	InferenceService [isvc-autogluon-v2-noslash] is Ready
21:40:52	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq	Unhealthy	Readiness probe failed: Get "http://10.134.0.36:8080/v2/models/isvc-autogluon-v2-noslash/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
21:40:54 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:40:54 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:40:55	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq	Killing	Stopping container kube-rbac-proxy
21:40:55	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq	Killing	Stopping container kserve-container
21:41:00	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-ts-v1-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-ts-v1-predictor-6cf79d9f7d from 0 to 1
21:41:01	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh	Created	Created container: storage-initializer
21:41:01	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh	Started	Started container storage-initializer
21:41:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1": the object has been modified; please apply your changes to the latest version and try again
21:41:01 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-ts-v1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1": the object has been modified; please apply your changes to the latest version and try again
21:41:01	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-ts-v1-predictor-6cf79d9f7d	SuccessfulCreate	Created pod: isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh
21:41:01	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
21:41:01	kserve-ci-e2e-test	multus	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh	AddedInterface	Add eth0 [10.134.0.37/23] from ovn-kubernetes
21:41:11	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh	Started	Started container kserve-container
21:41:11	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh	Started	Started container kube-rbac-proxy
21:41:11	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh	Created	Created container: kube-rbac-proxy
21:41:11	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:41:11	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh	Created	Created container: kserve-container
21:41:11	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
21:41:19 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh	Unhealthy	Readiness probe failed: dial tcp 10.134.0.37:8080: connect: connection refused
21:41:29 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:41:29	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	InferenceServiceReady	InferenceService [isvc-autogluon-ts-v1] is Ready
21:41:30 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:41:30 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:41:43	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh	Killing	Stopping container kube-rbac-proxy
21:41:43	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh	Killing	Stopping container kserve-container
21:41:44	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh	Unhealthy	Readiness probe failed: Get "https://10.134.0.37:8643/healthz": dial tcp 10.134.0.37:8643: connect: connection refused
21:41:48	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-ts-v1-noslash-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-ts-v1-noslash-predictor-b54fbf585 from 0 to 1
21:41:48	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
21:41:48	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585	SuccessfulCreate	Created pod: isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2
21:41:48	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-ts-v1-noslash": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1-noslash": the object has been modified; please apply your changes to the latest version and try again
21:41:49	kserve-ci-e2e-test	multus	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
21:41:49	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2	Started	Started container storage-initializer
21:41:49	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2	Created	Created container: storage-initializer
21:41:49	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
21:41:54	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2	Started	Started container kserve-container
21:41:54	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:41:54	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2	Started	Started container kube-rbac-proxy
21:41:54	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2	Created	Created container: kserve-container
21:41:54	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
21:41:54	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2	Created	Created container: kube-rbac-proxy
21:42:01 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2	Unhealthy	Readiness probe failed: dial tcp 10.134.0.38:8080: connect: connection refused
21:42:03	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:42:03	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:42:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	InferenceServiceReady	InferenceService [isvc-autogluon-ts-v1-noslash] is Ready
21:42:11 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1-noslash	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:42:18	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:42:18	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:42:23	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2	Killing	Stopping container kserve-container
21:42:23	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2	Killing	Stopping container kube-rbac-proxy
21:42:29	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-75c7994655-s4p7r	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
21:42:29	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7f74cfd478-88wlx	Created	Created container: storage-initializer
21:42:29	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-75c7994655-s4p7r	Started	Started container storage-initializer
21:42:29	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-75c7994655-s4p7r	Created	Created container: storage-initializer
21:42:29	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-create-v2-predictor-75c7994655 from 0 to 1
21:42:29	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again
21:42:29	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7f74cfd478-88wlx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
21:42:29	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	UpdateFailed	Failed to update status for InferenceService "isvc-canary-create": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again
21:42:29	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-predictor-7f74cfd478	SuccessfulCreate	Created pod: isvc-canary-create-predictor-7f74cfd478-88wlx
21:42:29	kserve-ci-e2e-test	deployment-controller	isvc-canary-create-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-create-predictor-7f74cfd478 from 0 to 1
21:42:29	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7f74cfd478-88wlx	Started	Started container storage-initializer
21:42:29	kserve-ci-e2e-test	replicaset-controller	isvc-canary-create-v2-predictor-75c7994655	SuccessfulCreate	Created pod: isvc-canary-create-v2-predictor-75c7994655-s4p7r
21:42:29	kserve-ci-e2e-test	multus	isvc-canary-create-v2-predictor-75c7994655-s4p7r	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
21:42:29	kserve-ci-e2e-test	multus	isvc-canary-create-predictor-7f74cfd478-88wlx	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
21:42:34	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7f74cfd478-88wlx	Started	Started container kube-rbac-proxy
21:42:34	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-75c7994655-s4p7r	Started	Started container kserve-container
21:42:34	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-75c7994655-s4p7r	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine
21:42:34	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7f74cfd478-88wlx	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:42:34	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7f74cfd478-88wlx	Created	Created container: kserve-container
21:42:34	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7f74cfd478-88wlx	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine
21:42:34	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-75c7994655-s4p7r	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:42:34	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-75c7994655-s4p7r	Created	Created container: kube-rbac-proxy
21:42:34	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7f74cfd478-88wlx	Created	Created container: kube-rbac-proxy
21:42:34	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-75c7994655-s4p7r	Started	Started container kube-rbac-proxy
21:42:34	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7f74cfd478-88wlx	Started	Started container kserve-container
21:42:34	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-75c7994655-s4p7r	Created	Created container: kserve-container
21:43:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	InferenceServiceReady	InferenceService [isvc-canary-create] is Ready
21:43:51 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-create	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:43:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-75c7994655-s4p7r	Killing	Stopping container kserve-container
21:43:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-75c7994655-s4p7r	Killing	Stopping container kube-rbac-proxy
21:43:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7f74cfd478-88wlx	Killing	Stopping container kube-rbac-proxy
21:43:59	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7f74cfd478-88wlx	Killing	Stopping container kserve-container
21:44:00 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7f74cfd478-88wlx	Unhealthy	Readiness probe failed: dial tcp 10.134.0.39:8080: connect: connection refused
21:44:01 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-75c7994655-s4p7r	Unhealthy	Readiness probe failed: dial tcp 10.134.0.40:8080: connect: connection refused
21:44:01	kserve-ci-e2e-test	kubelet	isvc-canary-create-v2-predictor-75c7994655-s4p7r	Unhealthy	Readiness probe failed: Get "https://10.134.0.40:8643/healthz": dial tcp 10.134.0.40:8643: connect: connection refused
21:44:01	kserve-ci-e2e-test	kubelet	isvc-canary-create-predictor-7f74cfd478-88wlx	Unhealthy	Readiness probe failed: Get "https://10.134.0.39:8643/healthz": dial tcp 10.134.0.39:8643: connect: connection refused
21:44:04	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq	Created	Created container: storage-initializer
21:44:04	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-v2-predictor-5ff9f946fc	SuccessfulCreate	Created pod: isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq
21:44:04	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-predictor-74874cc84b	SuccessfulCreate	Created pod: isvc-canary-promote-predictor-74874cc84b-7b45l
21:44:04	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-74874cc84b-7b45l	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
21:44:04	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-74874cc84b-7b45l	Created	Created container: storage-initializer
21:44:04	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-74874cc84b-7b45l	Started	Started container storage-initializer
21:44:04	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-predictor-74874cc84b from 0 to 1
21:44:04	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-v2-predictor-5ff9f946fc from 0 to 1
21:44:04	kserve-ci-e2e-test	multus	isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq	AddedInterface	Add eth0 [10.134.0.42/23] from ovn-kubernetes
21:44:04	kserve-ci-e2e-test	multus	isvc-canary-promote-predictor-74874cc84b-7b45l	AddedInterface	Add eth0 [10.134.0.41/23] from ovn-kubernetes
21:44:04	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq	Started	Started container storage-initializer
21:44:04	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
21:44:08	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:44:08	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq	Created	Created container: kube-rbac-proxy
21:44:08	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq	Started	Started container kube-rbac-proxy
21:44:08	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine
21:44:08	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq	Created	Created container: kserve-container
21:44:08	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq	Started	Started container kserve-container
21:44:09	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-74874cc84b-7b45l	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine
21:44:09	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-74874cc84b-7b45l	Started	Started container kube-rbac-proxy
21:44:09	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-74874cc84b-7b45l	Created	Created container: kube-rbac-proxy
21:44:09	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-74874cc84b-7b45l	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:44:09	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-74874cc84b-7b45l	Started	Started container kserve-container
21:44:09	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-74874cc84b-7b45l	Created	Created container: kserve-container
21:45:34	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-74874cc84b-7b45l	Killing	Stopping container kserve-container
21:45:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	InferenceServiceNotReady	InferenceService [isvc-canary-promote] is no longer Ready because of: PredictorReady, Ready, Stopped
21:45:34	kserve-ci-e2e-test	replicaset-controller	isvc-canary-promote-v2-predictor-5ff9f946fc	SuccessfulCreate	Created pod: isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd
21:45:34	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-74874cc84b-7b45l	Killing	Stopping container kube-rbac-proxy
21:45:34	kserve-ci-e2e-test	deployment-controller	isvc-canary-promote-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-promote-v2-predictor-5ff9f946fc from 1 to 2
21:45:34 (x19)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:45:34 (x3)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	UpdateFailed	Failed to update status for InferenceService "isvc-canary-promote": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again
21:45:34 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again
21:45:35	kserve-ci-e2e-test	multus	isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd	AddedInterface	Add eth0 [10.133.0.22/23] from ovn-kubernetes
21:45:35	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894"
21:45:36	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-74874cc84b-7b45l	Unhealthy	Readiness probe failed: Get "https://10.134.0.41:8643/healthz": dial tcp 10.134.0.41:8643: connect: connection refused
21:45:36 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-predictor-74874cc84b-7b45l	Unhealthy	Readiness probe failed: dial tcp 10.134.0.41:8080: connect: connection refused
21:45:38	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd	Started	Started container storage-initializer
21:45:38	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd	Created	Created container: storage-initializer
21:45:38	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" in 3.786s (3.786s including waiting). Image size: 302292931 bytes.
21:45:42	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716"
21:45:55	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
21:45:55	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd	Created	Created container: kserve-container
21:45:55	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" in 12.777s (12.777s including waiting). Image size: 1557885864 bytes.
21:45:55	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd	Started	Started container kserve-container
21:45:57	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.236s (2.236s including waiting). Image size: 211946088 bytes.
21:45:58	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd	Started	Started container kube-rbac-proxy
21:45:58	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd	Created	Created container: kube-rbac-proxy
21:47:15 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-promote	InferenceServiceReady	InferenceService [isvc-canary-promote] is Ready
21:47:24	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq	Killing	Stopping container kserve-container
21:47:24	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq	Killing	Stopping container kube-rbac-proxy
21:47:24	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd	Killing	Stopping container kserve-container
21:47:24	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd	Killing	Stopping container kube-rbac-proxy
21:47:25 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq	Unhealthy	Readiness probe failed: dial tcp 10.134.0.42:8080: connect: connection refused
21:47:25 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd	Unhealthy	Readiness probe failed: dial tcp 10.133.0.22:8080: connect: connection refused
21:47:25	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd	Unhealthy	Readiness probe failed: Get "https://10.133.0.22:8643/healthz": dial tcp 10.133.0.22:8643: connect: connection refused
21:47:25	kserve-ci-e2e-test	kubelet	isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq	Unhealthy	Readiness probe failed: Get "https://10.134.0.42:8643/healthz": dial tcp 10.134.0.42:8643: connect: connection refused
21:47:29	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-predictor-6d775fdd5d from 0 to 1
21:47:29	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-6d775fdd5d	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-6d775fdd5d-ll64b
21:47:29	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-v2-predictor-5b9f7db4c5	SuccessfulCreate	Created pod: isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2
21:47:29	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-v2-predictor-5b9f7db4c5 from 0 to 1
21:47:30	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
21:47:30	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-ll64b	Created	Created container: storage-initializer
21:47:30	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2	Created	Created container: storage-initializer
21:47:30	kserve-ci-e2e-test	multus	isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2	AddedInterface	Add eth0 [10.133.0.24/23] from ovn-kubernetes
21:47:30	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2	Started	Started container storage-initializer
21:47:30	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-6d775fdd5d-ll64b	AddedInterface	Add eth0 [10.133.0.23/23] from ovn-kubernetes
21:47:30	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-ll64b	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
21:47:30	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-ll64b	Started	Started container storage-initializer
21:47:34	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2	Created	Created container: kube-rbac-proxy
21:47:34	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:47:34	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-ll64b	Created	Created container: kserve-container
21:47:34	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-ll64b	Started	Started container kserve-container
21:47:34	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2	Started	Started container kserve-container
21:47:34	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-ll64b	Created	Created container: kube-rbac-proxy
21:47:34	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-ll64b	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:47:34	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2	Started	Started container kube-rbac-proxy
21:47:34	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2	Created	Created container: kserve-container
21:47:34	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine
21:47:34	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-ll64b	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine
21:47:34	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-ll64b	Started	Started container kube-rbac-proxy
21:48:59	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2	Killing	Stopping container kserve-container
21:48:59	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2	Killing	Stopping container kube-rbac-proxy
21:48:59	kserve-ci-e2e-test	deployment-controller	isvc-canary-rollback-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-rollback-predictor-6d775fdd5d from 1 to 2
21:48:59	kserve-ci-e2e-test	replicaset-controller	isvc-canary-rollback-predictor-6d775fdd5d	SuccessfulCreate	Created pod: isvc-canary-rollback-predictor-6d775fdd5d-l5l4f
21:48:59 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	UpdateFailed	Failed to update status for InferenceService "isvc-canary-rollback": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again
21:48:59 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again
21:49:00	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-l5l4f	Started	Started container storage-initializer
21:49:00	kserve-ci-e2e-test	multus	isvc-canary-rollback-predictor-6d775fdd5d-l5l4f	AddedInterface	Add eth0 [10.134.0.43/23] from ovn-kubernetes
21:49:00	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-l5l4f	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
21:49:00	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-l5l4f	Created	Created container: storage-initializer
21:49:00 (x20)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:49:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	InferenceServiceNotReady	InferenceService [isvc-canary-rollback] is no longer Ready because of: PredictorReady, Ready, Stopped
21:49:02 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2	Unhealthy	Readiness probe failed: dial tcp 10.133.0.24:8080: connect: connection refused
21:49:02	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2	Unhealthy	Readiness probe failed: Get "https://10.133.0.24:8643/healthz": dial tcp 10.133.0.24:8643: connect: connection refused
21:49:04	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-l5l4f	Started	Started container kserve-container
21:49:04	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-l5l4f	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine
21:49:04	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-l5l4f	Created	Created container: kserve-container
21:49:04	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-l5l4f	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:49:04	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-l5l4f	Created	Created container: kube-rbac-proxy
21:49:04	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-l5l4f	Started	Started container kube-rbac-proxy
21:50:22 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-rollback	InferenceServiceReady	InferenceService [isvc-canary-rollback] is Ready
21:50:30	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-ll64b	Killing	Stopping container kube-rbac-proxy
21:50:30	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-ll64b	Killing	Stopping container kserve-container
21:50:30	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-l5l4f	Killing	Stopping container kserve-container
21:50:30	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-l5l4f	Killing	Stopping container kube-rbac-proxy
21:50:31	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-ll64b	Unhealthy	Readiness probe failed: Get "https://10.133.0.23:8643/healthz": dial tcp 10.133.0.23:8643: connect: connection refused
21:50:31 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-ll64b	Unhealthy	Readiness probe failed: dial tcp 10.133.0.23:8080: connect: connection refused
21:50:32 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-l5l4f	Unhealthy	Readiness probe failed: dial tcp 10.134.0.43:8080: connect: connection refused
21:50:32	kserve-ci-e2e-test	kubelet	isvc-canary-rollback-predictor-6d775fdd5d-l5l4f	Unhealthy	Readiness probe failed: Get "https://10.134.0.43:8643/healthz": dial tcp 10.134.0.43:8643: connect: connection refused
21:50:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	UpdateFailed	Failed to update status for InferenceService "isvc-canary-stop": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again
21:50:35	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-v2-predictor-5b66dd5859	SuccessfulCreate	Created pod: isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd
21:50:35	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-stop-v2-predictor-5b66dd5859 from 0 to 1
21:50:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-8bd7d86fd-jwvtx	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-stop-predictor-serving-cert" not found
21:50:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again
21:50:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd	Started	Started container storage-initializer
21:50:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd	Created	Created container: storage-initializer
21:50:35	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
21:50:35	kserve-ci-e2e-test	multus	isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd	AddedInterface	Add eth0 [10.133.0.26/23] from ovn-kubernetes
21:50:35	kserve-ci-e2e-test	deployment-controller	isvc-canary-stop-predictor	ScalingReplicaSet	Scaled up replica set isvc-canary-stop-predictor-8bd7d86fd from 0 to 1
21:50:35	kserve-ci-e2e-test	replicaset-controller	isvc-canary-stop-predictor-8bd7d86fd	SuccessfulCreate	Created pod: isvc-canary-stop-predictor-8bd7d86fd-jwvtx
21:50:36	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-8bd7d86fd-jwvtx	Started	Started container storage-initializer
21:50:36	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-8bd7d86fd-jwvtx	Created	Created container: storage-initializer
21:50:36	kserve-ci-e2e-test	multus	isvc-canary-stop-predictor-8bd7d86fd-jwvtx	AddedInterface	Add eth0 [10.133.0.25/23] from ovn-kubernetes
21:50:36	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-8bd7d86fd-jwvtx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
21:50:40	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:50:40	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine
21:50:40	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd	Created	Created container: kserve-container
21:50:40	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd	Started	Started container kserve-container
21:50:40	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd	Started	Started container kube-rbac-proxy
21:50:40	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd	Created	Created container: kube-rbac-proxy
21:50:41	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-8bd7d86fd-jwvtx	Created	Created container: kserve-container
21:50:41	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-8bd7d86fd-jwvtx	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine
21:50:41	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-8bd7d86fd-jwvtx	Started	Started container kserve-container
21:50:41	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-8bd7d86fd-jwvtx	Started	Started container kube-rbac-proxy
21:50:41	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-8bd7d86fd-jwvtx	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:50:41	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-8bd7d86fd-jwvtx	Created	Created container: kube-rbac-proxy
21:51:59	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	InferenceServiceReady	InferenceService [isvc-canary-stop] is Ready
21:52:05	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-8bd7d86fd-jwvtx	Killing	Stopping container kserve-container
21:52:05	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd	Killing	Stopping container kube-rbac-proxy
21:52:05 (x20)	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:52:05	kserve-ci-e2e-test	v1beta1Controllers	isvc-canary-stop	InferenceServiceNotReady	InferenceService [isvc-canary-stop] is no longer Ready because of: PredictorReady, Ready
21:52:05	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd	Killing	Stopping container kserve-container
21:52:05	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-8bd7d86fd-jwvtx	Killing	Stopping container kube-rbac-proxy
21:52:07 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd	Unhealthy	Readiness probe failed: dial tcp 10.133.0.26:8080: connect: connection refused
21:52:07	kserve-ci-e2e-test	kubelet	isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd	Unhealthy	Readiness probe failed: Get "https://10.133.0.26:8643/healthz": dial tcp 10.133.0.26:8643: connect: connection refused
21:52:09	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-8bd7d86fd-jwvtx	Unhealthy	Readiness probe failed: Get "https://10.133.0.25:8643/healthz": dial tcp 10.133.0.25:8643: connect: connection refused
21:52:09 (x10)	kserve-ci-e2e-test	kubelet	isvc-canary-stop-predictor-8bd7d86fd-jwvtx	Unhealthy	Readiness probe failed: dial tcp 10.133.0.25:8080: connect: connection refused
21:52:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm": the object has been modified; please apply your changes to the latest version and try again
21:52:15	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-predictor-54f85ddf88	SuccessfulCreate	Created pod: isvc-lightgbm-predictor-54f85ddf88-kndbs
21:52:15 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm": the object has been modified; please apply your changes to the latest version and try again
21:52:15	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-kndbs	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-lightgbm-predictor-serving-cert" not found
21:52:15	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-predictor-54f85ddf88 from 0 to 1
21:52:16	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-kndbs	Started	Started container storage-initializer
21:52:16	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-kndbs	Created	Created container: storage-initializer
21:52:16	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-kndbs	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
21:52:16	kserve-ci-e2e-test	multus	isvc-lightgbm-predictor-54f85ddf88-kndbs	AddedInterface	Add eth0 [10.133.0.27/23] from ovn-kubernetes
21:52:21	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-kndbs	Pulling	Pulling image "kserve/lgbserver:latest"
21:52:27	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-kndbs	Started	Started container kserve-container
21:52:27	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-kndbs	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:52:27	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-kndbs	Created	Created container: kserve-container
21:52:27	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-kndbs	Pulled	Successfully pulled image "kserve/lgbserver:latest" in 6.497s (6.497s including waiting). Image size: 607170828 bytes.
21:52:28	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-kndbs	Started	Started container kube-rbac-proxy
21:52:28	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-kndbs	Created	Created container: kube-rbac-proxy
21:53:00 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:53:00 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:53:35 (x9)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-kndbs	Unhealthy	Readiness probe failed: dial tcp 10.133.0.27:8080: connect: connection refused
21:53:45 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:53:45 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:53:45 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:53:45	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	InferenceServiceReady	InferenceService [isvc-lightgbm] is Ready
21:53:55	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-kndbs	Killing	Stopping container kube-rbac-proxy
21:53:55	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-kndbs	Killing	Stopping container kserve-container
21:54:00	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-runtime-predictor-577f7db7cb from 0 to 1
21:54:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-runtime": the object has been modified; please apply your changes to the latest version and try again
21:54:00 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-runtime": the object has been modified; please apply your changes to the latest version and try again
21:54:00	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-runtime-predictor-577f7db7cb	SuccessfulCreate	Created pod: isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm
21:54:01	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm	Started	Started container storage-initializer
21:54:01	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
21:54:01	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm	Created	Created container: storage-initializer
21:54:01	kserve-ci-e2e-test	multus	isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm	AddedInterface	Add eth0 [10.133.0.28/23] from ovn-kubernetes
21:54:01	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-54f85ddf88-kndbs	Unhealthy	Readiness probe failed: Get "https://10.133.0.27:8643/healthz": context deadline exceeded
21:54:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm	Created	Created container: kube-rbac-proxy
21:54:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm	Pulled	Container image "kserve/lgbserver:latest" already present on machine
21:54:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm	Created	Created container: kserve-container
21:54:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm	Started	Started container kserve-container
21:54:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:54:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm	Started	Started container kube-rbac-proxy
21:54:45 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:54:45 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:55:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-runtime] is Ready
21:55:23 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:55:30 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:55:30 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:55:31	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm	Killing	Stopping container kserve-container
21:55:31	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm	Killing	Stopping container kube-rbac-proxy
21:55:32	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm	Unhealthy	Readiness probe failed: Get "https://10.133.0.28:8643/healthz": dial tcp 10.133.0.28:8643: connect: connection refused
21:55:32 (x10)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm	Unhealthy	Readiness probe failed: dial tcp 10.133.0.28:8080: connect: connection refused
21:55:36	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
21:55:36 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
21:55:36	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5	SuccessfulCreate	Created pod: isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn
21:55:36	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-runtime-predictor-7f4957c5d5 from 0 to 1
21:55:37	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn	Started	Started container storage-initializer
21:55:37	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn	AddedInterface	Add eth0 [10.133.0.29/23] from ovn-kubernetes
21:55:37	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn	Created	Created container: storage-initializer
21:55:37	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
21:55:42	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn	Pulling	Pulling image "docker.io/seldonio/mlserver:1.7.1"
21:57:42	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:57:42	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn	Pulled	Successfully pulled image "docker.io/seldonio/mlserver:1.7.1" in 2m0.321s (2m0.321s including waiting). Image size: 10890461297 bytes.
21:57:42	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn	Created	Created container: kube-rbac-proxy
21:57:42	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn	Started	Started container kserve-container
21:57:42	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn	Started	Started container kube-rbac-proxy
21:57:42	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn	Created	Created container: kserve-container
21:58:06 (x10)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:58:06 (x10)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:58:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-runtime] is Ready
21:58:19 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:58:21	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:58:21	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:58:26	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn	Killing	Stopping container kube-rbac-proxy
21:58:26	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn	Killing	Stopping container kserve-container
21:58:31	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5 from 0 to 1
21:58:32	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
21:58:32	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5	SuccessfulCreate	Created pod: isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968
21:58:32 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
21:58:32	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968	AddedInterface	Add eth0 [10.133.0.30/23] from ovn-kubernetes
21:58:32	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
21:58:32	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968	Created	Created container: storage-initializer
21:58:32	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968	Started	Started container storage-initializer
21:58:36	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968	Pulled	Container image "kserve/lgbserver:latest" already present on machine
21:58:36	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968	Created	Created container: kserve-container
21:58:36	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:58:36	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968	Started	Started container kserve-container
21:58:37	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968	Started	Started container kube-rbac-proxy
21:58:37	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968	Created	Created container: kube-rbac-proxy
21:58:43 (x3)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968	Unhealthy	Readiness probe failed: dial tcp 10.133.0.30:8080: connect: connection refused
21:58:47	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:58:47	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:58:54 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:58:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-kserve] is Ready
21:59:02	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:59:02	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:59:02	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968	Killing	Stopping container kube-rbac-proxy
21:59:02	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968	Killing	Stopping container kserve-container
21:59:07	kserve-ci-e2e-test	multus	isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl	AddedInterface	Add eth0 [10.133.0.31/23] from ovn-kubernetes
21:59:07	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
21:59:07	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl	Created	Created container: storage-initializer
21:59:07	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl	Started	Started container storage-initializer
21:59:07	kserve-ci-e2e-test	replicaset-controller	isvc-mlflow-v2-runtime-predictor-6456f94494	SuccessfulCreate	Created pod: isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl
21:59:07	kserve-ci-e2e-test	deployment-controller	isvc-mlflow-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-mlflow-v2-runtime-predictor-6456f94494 from 0 to 1
21:59:07	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-mlflow-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-mlflow-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
21:59:12	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:59:12	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl	Created	Created container: kube-rbac-proxy
21:59:12	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl	Started	Started container kube-rbac-proxy
21:59:12	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl	Started	Started container kserve-container
21:59:12	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
21:59:12	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl	Created	Created container: kserve-container
21:59:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	InferenceServiceReady	InferenceService [isvc-mlflow-v2-runtime] is Ready
21:59:49 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:59:52 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:59:52 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:59:57	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl	Killing	Stopping container kserve-container
21:59:57	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl	Killing	Stopping container kube-rbac-proxy
21:59:58	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl	Unhealthy	Readiness probe failed: Get "https://10.133.0.31:8643/healthz": dial tcp 10.133.0.31:8643: connect: connection refused
22:00:02	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-mcp-predictor-5cf4b5779c	SuccessfulCreate	Created pod: isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv
22:00:02	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-mcp-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-mcp-predictor-5cf4b5779c from 0 to 1
22:00:02	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-mcp": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-mcp": the object has been modified; please apply your changes to the latest version and try again
22:00:02	kserve-ci-e2e-test	multus	isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv	AddedInterface	Add eth0 [10.133.0.32/23] from ovn-kubernetes
22:00:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:00:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv	Started	Started container storage-initializer
22:00:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv	Created	Created container: storage-initializer
22:00:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv	Created	Created container: kserve-container
22:00:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv	Pulling	Pulling image "quay.io/opendatahub/kserve-agent:latest"
22:00:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv	Started	Started container kserve-container
22:00:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine
22:00:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv	Created	Created container: kserve-agent
22:00:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent:latest" in 2.635s (2.635s including waiting). Image size: 239481897 bytes.
22:00:10	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv	Started	Started container kserve-agent
22:00:10	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:00:10	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv	Started	Started container kube-rbac-proxy
22:00:10	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv	Created	Created container: kube-rbac-proxy
22:01:16 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:01:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	InferenceServiceReady	InferenceService [isvc-sklearn-mcp] is Ready
22:01:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv	Killing	Stopping container kube-rbac-proxy
22:01:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv	Killing	Stopping container kserve-agent
22:01:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv	Killing	Stopping container kserve-container
22:01:27	kserve-ci-e2e-test	deployment-controller	isvc-paddle-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-predictor-6d5c9fd4f6 from 0 to 1
22:01:27	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-predictor-6d5c9fd4f6	SuccessfulCreate	Created pod: isvc-paddle-predictor-6d5c9fd4f6-fcrw2
22:01:27	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	UpdateFailed	Failed to update status for InferenceService "isvc-paddle": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle": the object has been modified; please apply your changes to the latest version and try again
22:01:28	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-fcrw2	Created	Created container: storage-initializer
22:01:28	kserve-ci-e2e-test	multus	isvc-paddle-predictor-6d5c9fd4f6-fcrw2	AddedInterface	Add eth0 [10.134.0.44/23] from ovn-kubernetes
22:01:28	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-fcrw2	Started	Started container storage-initializer
22:01:28	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-fcrw2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:01:34	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-fcrw2	Pulling	Pulling image "kserve/paddleserver:latest"
22:01:46	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-fcrw2	Pulled	Successfully pulled image "kserve/paddleserver:latest" in 12.558s (12.558s including waiting). Image size: 1163606776 bytes.
22:01:46 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv	Unhealthy	Readiness probe failed: Get "http://10.133.0.32:8080/v1/models/isvc-sklearn-mcp": dial tcp 10.133.0.32:8080: connect: connection refused
22:01:46	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-fcrw2	Started	Started container kube-rbac-proxy
22:01:46	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-fcrw2	Created	Created container: kube-rbac-proxy
22:01:46	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-fcrw2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:01:46	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-fcrw2	Started	Started container kserve-container
22:01:46	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-fcrw2	Created	Created container: kserve-container
22:01:51 (x6)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv	Unhealthy	Readiness probe failed: Get "https://10.133.0.32:8643/healthz": dial tcp 10.133.0.32:8643: connect: connection refused
22:02:12 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:02:12 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:02:23 (x6)	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-fcrw2	Unhealthy	Readiness probe failed: dial tcp 10.134.0.44:8080: connect: connection refused
22:02:27	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:02:27	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:02:33 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:02:33	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	InferenceServiceReady	InferenceService [isvc-paddle] is Ready
22:02:39	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-fcrw2	Killing	Stopping container kserve-container
22:02:39	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6d5c9fd4f6-fcrw2	Killing	Stopping container kube-rbac-proxy
22:02:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-runtime": the object has been modified; please apply your changes to the latest version and try again
22:02:44	kserve-ci-e2e-test	deployment-controller	isvc-paddle-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-runtime-predictor-58b99fff76 from 0 to 1
22:02:44	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-runtime-predictor-58b99fff76	SuccessfulCreate	Created pod: isvc-paddle-runtime-predictor-58b99fff76-v9vk8
22:02:45	kserve-ci-e2e-test	multus	isvc-paddle-runtime-predictor-58b99fff76-v9vk8	AddedInterface	Add eth0 [10.133.0.33/23] from ovn-kubernetes
22:02:45	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-v9vk8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:02:45	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-v9vk8	Created	Created container: storage-initializer
22:02:45	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-v9vk8	Started	Started container storage-initializer
22:02:50	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-v9vk8	Pulling	Pulling image "kserve/paddleserver:latest"
22:03:02	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-v9vk8	Pulled	Successfully pulled image "kserve/paddleserver:latest" in 11.415s (11.415s including waiting). Image size: 1163606776 bytes.
22:03:02	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-v9vk8	Started	Started container kserve-container
22:03:02	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-v9vk8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:03:02	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-v9vk8	Created	Created container: kube-rbac-proxy
22:03:02	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-v9vk8	Started	Started container kube-rbac-proxy
22:03:02	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-v9vk8	Created	Created container: kserve-container
22:03:29 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:03:29 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:03:38 (x6)	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-v9vk8	Unhealthy	Readiness probe failed: dial tcp 10.133.0.33:8080: connect: connection refused
22:03:44	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:03:44	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:03:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	InferenceServiceReady	InferenceService [isvc-paddle-runtime] is Ready
22:03:49 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:03:56	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-v9vk8	Killing	Stopping container kserve-container
22:03:56	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-58b99fff76-v9vk8	Killing	Stopping container kube-rbac-proxy
22:04:01	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2	Started	Started container storage-initializer
22:04:01	kserve-ci-e2e-test	multus	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2	AddedInterface	Add eth0 [10.134.0.45/23] from ovn-kubernetes
22:04:01	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:04:01	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2	Created	Created container: storage-initializer
22:04:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
22:04:01	kserve-ci-e2e-test	deployment-controller	isvc-paddle-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-v2-kserve-predictor-64d4c74b4d from 0 to 1
22:04:01	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-v2-kserve-predictor-64d4c74b4d	SuccessfulCreate	Created pod: isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2
22:04:06	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2	Created	Created container: kube-rbac-proxy
22:04:06	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:04:06	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2	Started	Started container kserve-container
22:04:06	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2	Started	Started container kube-rbac-proxy
22:04:06	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2	Created	Created container: kserve-container
22:04:06	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2	Pulled	Container image "kserve/paddleserver:latest" already present on machine
22:04:16	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:04:16	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:04:53 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:04:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	InferenceServiceReady	InferenceService [isvc-paddle-v2-kserve] is Ready
22:05:01 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:05:01 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:05:03	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2	Unhealthy	Readiness probe failed: Get "https://10.134.0.45:8643/healthz": dial tcp 10.134.0.45:8643: connect: connection refused
22:05:03 (x7)	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2	Unhealthy	Readiness probe failed: dial tcp 10.134.0.45:8080: connect: connection refused
22:05:03	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2	Killing	Stopping container kserve-container
22:05:03	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2	Killing	Stopping container kube-rbac-proxy
22:05:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	UpdateFailed	Failed to update status for InferenceService "isvc-pmml": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml": the object has been modified; please apply your changes to the latest version and try again
22:05:08	kserve-ci-e2e-test	deployment-controller	isvc-pmml-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-predictor-8674795f59 from 0 to 1
22:05:08	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-cvmzv	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-pmml-predictor-serving-cert" not found
22:05:08	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-predictor-8674795f59	SuccessfulCreate	Created pod: isvc-pmml-predictor-8674795f59-cvmzv
22:05:09	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-cvmzv	Created	Created container: storage-initializer
22:05:09	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-cvmzv	Started	Started container storage-initializer
22:05:09	kserve-ci-e2e-test	multus	isvc-pmml-predictor-8674795f59-cvmzv	AddedInterface	Add eth0 [10.133.0.34/23] from ovn-kubernetes
22:05:09	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-cvmzv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:05:14	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-cvmzv	Pulling	Pulling image "kserve/pmmlserver:latest"
22:05:21	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-cvmzv	Created	Created container: kube-rbac-proxy
22:05:21	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-cvmzv	Pulled	Successfully pulled image "kserve/pmmlserver:latest" in 6.916s (6.916s including waiting). Image size: 801937285 bytes.
22:05:21	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-cvmzv	Started	Started container kube-rbac-proxy
22:05:21	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-cvmzv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:05:21	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-cvmzv	Started	Started container kserve-container
22:05:21	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-cvmzv	Created	Created container: kserve-container
22:05:38 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:05:38 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:06:28 (x9)	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-cvmzv	Unhealthy	Readiness probe failed: dial tcp 10.133.0.34:8080: connect: connection refused
22:06:36 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:06:36	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	InferenceServiceReady	InferenceService [isvc-pmml] is Ready
22:06:38 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:06:38 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:06:39	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-cvmzv	Killing	Stopping container kserve-container
22:06:39	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8674795f59-cvmzv	Killing	Stopping container kube-rbac-proxy
22:06:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-runtime": the object has been modified; please apply your changes to the latest version and try again
22:06:44 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-runtime": the object has been modified; please apply your changes to the latest version and try again
22:06:44	kserve-ci-e2e-test	deployment-controller	isvc-pmml-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-runtime-predictor-885fc8777 from 0 to 1
22:06:44	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-runtime-predictor-885fc8777	SuccessfulCreate	Created pod: isvc-pmml-runtime-predictor-885fc8777-nmh2s
22:06:45	kserve-ci-e2e-test	multus	isvc-pmml-runtime-predictor-885fc8777-nmh2s	AddedInterface	Add eth0 [10.133.0.35/23] from ovn-kubernetes
22:06:45	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-nmh2s	Created	Created container: storage-initializer
22:06:45	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-nmh2s	Started	Started container storage-initializer
22:06:45	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-nmh2s	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:06:49	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-nmh2s	Created	Created container: kube-rbac-proxy
22:06:49	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-nmh2s	Started	Started container kube-rbac-proxy
22:06:49	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-nmh2s	Started	Started container kserve-container
22:06:49	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-nmh2s	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:06:49	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-nmh2s	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
22:06:49	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-nmh2s	Created	Created container: kserve-container
22:07:14 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:07:14 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:07:56 (x9)	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-nmh2s	Unhealthy	Readiness probe failed: dial tcp 10.133.0.35:8080: connect: connection refused
22:07:59 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:07:59 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:08:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	InferenceServiceReady	InferenceService [isvc-pmml-runtime] is Ready
22:08:04 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:08:06	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-nmh2s	Killing	Stopping container kube-rbac-proxy
22:08:06	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-nmh2s	Killing	Stopping container kserve-container
22:08:06	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-885fc8777-nmh2s	Unhealthy	Readiness probe failed: Get "https://10.133.0.35:8643/healthz": dial tcp 10.133.0.35:8643: connect: connection refused
22:08:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
22:08:10 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
22:08:10	kserve-ci-e2e-test	deployment-controller	isvc-pmml-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-v2-kserve-predictor-84cb55486c from 0 to 1
22:08:10	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-v2-kserve-predictor-84cb55486c	SuccessfulCreate	Created pod: isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt
22:08:11	kserve-ci-e2e-test	multus	isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt	AddedInterface	Add eth0 [10.133.0.36/23] from ovn-kubernetes
22:08:11	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt	Started	Started container storage-initializer
22:08:11	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt	Created	Created container: storage-initializer
22:08:11	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:08:16	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt	Created	Created container: kube-rbac-proxy
22:08:16	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:08:16	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt	Started	Started container kserve-container
22:08:16	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt	Created	Created container: kserve-container
22:08:16	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt	Started	Started container kube-rbac-proxy
22:08:16	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
22:08:40 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:08:40 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:09:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	InferenceServiceReady	InferenceService [isvc-pmml-v2-kserve] is Ready
22:09:34 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:09:41 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:09:41 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:09:42	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt	Killing	Stopping container kserve-container
22:09:42	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt	Killing	Stopping container kube-rbac-proxy
22:09:44 (x11)	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt	Unhealthy	Readiness probe failed: dial tcp 10.133.0.36:8080: connect: connection refused
22:09:44	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt	Unhealthy	Readiness probe failed: Get "https://10.133.0.36:8643/healthz": dial tcp 10.133.0.36:8643: connect: connection refused
22:09:47	kserve-ci-e2e-test	replicaset-controller	isvc-primary-5c6eb3-predictor-5587458446	SuccessfulCreate	Created pod: isvc-primary-5c6eb3-predictor-5587458446-fllrp
22:09:47	kserve-ci-e2e-test	deployment-controller	isvc-primary-5c6eb3-predictor	ScalingReplicaSet	Scaled up replica set isvc-primary-5c6eb3-predictor-5587458446 from 0 to 1
22:09:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-5c6eb3	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-5c6eb3": the object has been modified; please apply your changes to the latest version and try again
22:09:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-5c6eb3	UpdateFailed	Failed to update status for InferenceService "isvc-primary-5c6eb3": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-5c6eb3": the object has been modified; please apply your changes to the latest version and try again
22:09:47	kserve-ci-e2e-test	kubelet	isvc-primary-5c6eb3-predictor-5587458446-fllrp	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-primary-5c6eb3-predictor-serving-cert" not found
22:09:48	kserve-ci-e2e-test	kubelet	isvc-primary-5c6eb3-predictor-5587458446-fllrp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:09:48	kserve-ci-e2e-test	kubelet	isvc-primary-5c6eb3-predictor-5587458446-fllrp	Created	Created container: storage-initializer
22:09:48	kserve-ci-e2e-test	multus	isvc-primary-5c6eb3-predictor-5587458446-fllrp	AddedInterface	Add eth0 [10.133.0.37/23] from ovn-kubernetes
22:09:48	kserve-ci-e2e-test	kubelet	isvc-primary-5c6eb3-predictor-5587458446-fllrp	Started	Started container storage-initializer
22:09:52	kserve-ci-e2e-test	kubelet	isvc-primary-5c6eb3-predictor-5587458446-fllrp	Started	Started container kube-rbac-proxy
22:09:52	kserve-ci-e2e-test	kubelet	isvc-primary-5c6eb3-predictor-5587458446-fllrp	Created	Created container: kube-rbac-proxy
22:09:52	kserve-ci-e2e-test	kubelet	isvc-primary-5c6eb3-predictor-5587458446-fllrp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:09:52	kserve-ci-e2e-test	kubelet	isvc-primary-5c6eb3-predictor-5587458446-fllrp	Started	Started container kserve-container
22:09:52	kserve-ci-e2e-test	kubelet	isvc-primary-5c6eb3-predictor-5587458446-fllrp	Created	Created container: kserve-container
22:09:52	kserve-ci-e2e-test	kubelet	isvc-primary-5c6eb3-predictor-5587458446-fllrp	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine
22:10:02	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-5c6eb3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:10:02	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-5c6eb3-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:11:00 (x9)	kserve-ci-e2e-test	kubelet	isvc-primary-5c6eb3-predictor-5587458446-fllrp	Unhealthy	Readiness probe failed: dial tcp 10.133.0.37:8080: connect: connection refused
22:11:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-5c6eb3	InferenceServiceReady	InferenceService [isvc-primary-5c6eb3] is Ready
22:11:10 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-5c6eb3	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:11:32 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-5c6eb3-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:11:32	kserve-ci-e2e-test	deployment-controller	isvc-secondary-5c6eb3-predictor	ScalingReplicaSet	Scaled up replica set isvc-secondary-5c6eb3-predictor-7745f9d6b5 from 0 to 1
22:11:32	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-5c6eb3	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-5c6eb3": the object has been modified; please apply your changes to the latest version and try again
22:11:32	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-5c6eb3	UpdateFailed	Failed to update status for InferenceService "isvc-secondary-5c6eb3": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-5c6eb3": the object has been modified; please apply your changes to the latest version and try again
22:11:32 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-5c6eb3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:11:32	kserve-ci-e2e-test	replicaset-controller	isvc-secondary-5c6eb3-predictor-7745f9d6b5	SuccessfulCreate	Created pod: isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz
22:11:33	kserve-ci-e2e-test	multus	isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz	AddedInterface	Add eth0 [10.133.0.38/23] from ovn-kubernetes
22:11:39 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:11:39 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz	Created	Created container: storage-initializer
22:11:39 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz	Started	Started container storage-initializer
22:11:45	kserve-ci-e2e-test	kubelet	isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz	BackOff	Back-off restarting failed container storage-initializer in pod isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz_kserve-ci-e2e-test(de3d0eaa-76c8-478d-ba9d-efd09407cde7)
22:11:46 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-5c6eb3	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:11:47	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-5c6eb3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:11:47	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-5c6eb3-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:11:50	kserve-ci-e2e-test	kubelet	isvc-primary-5c6eb3-predictor-5587458446-fllrp	Killing	Stopping container kserve-container
22:11:50	kserve-ci-e2e-test	kubelet	isvc-primary-5c6eb3-predictor-5587458446-fllrp	Killing	Stopping container kube-rbac-proxy
22:11:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-baf50e	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-baf50e": the object has been modified; please apply your changes to the latest version and try again
22:11:55	kserve-ci-e2e-test	deployment-controller	isvc-init-fail-baf50e-predictor	ScalingReplicaSet	Scaled up replica set isvc-init-fail-baf50e-predictor-5dcc4f6f45 from 0 to 1
22:11:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-baf50e	UpdateFailed	Failed to update status for InferenceService "isvc-init-fail-baf50e": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-baf50e": the object has been modified; please apply your changes to the latest version and try again
22:11:55	kserve-ci-e2e-test	replicaset-controller	isvc-init-fail-baf50e-predictor-5dcc4f6f45	SuccessfulCreate	Created pod: isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26
22:11:56	kserve-ci-e2e-test	multus	isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26	AddedInterface	Add eth0 [10.133.0.39/23] from ovn-kubernetes
22:12:01 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:12:01 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26	Created	Created container: storage-initializer
22:12:01 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26	Started	Started container storage-initializer
22:12:02 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-baf50e	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:12:05	kserve-ci-e2e-test	kubelet	isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26	Killing	Stopping container storage-initializer
22:12:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn": the object has been modified; please apply your changes to the latest version and try again
22:12:11	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-predictor-5f849d69df-z4q42	AddedInterface	Add eth0 [10.133.0.40/23] from ovn-kubernetes
22:12:11	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-z4q42	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:12:11	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-z4q42	Created	Created container: storage-initializer
22:12:11	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-z4q42	Started	Started container storage-initializer
22:12:11	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-predictor-5f849d69df from 0 to 1
22:12:11	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-predictor-5f849d69df	SuccessfulCreate	Created pod: isvc-predictive-sklearn-predictor-5f849d69df-z4q42
22:12:15	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-z4q42	Pulling	Pulling image "kserve/predictiveserver:latest"
22:12:36	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-z4q42	Created	Created container: kserve-container
22:12:36	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-z4q42	Started	Started container kserve-container
22:12:36	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-z4q42	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:12:36	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-z4q42	Pulled	Successfully pulled image "kserve/predictiveserver:latest" in 20.861s (20.861s including waiting). Image size: 2353326736 bytes.
22:12:37	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-z4q42	Created	Created container: kube-rbac-proxy
22:12:37	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-z4q42	Started	Started container kube-rbac-proxy
22:13:11 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:13:11 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:13:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	InferenceServiceReady	InferenceService [isvc-predictive-sklearn] is Ready
22:13:54 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:13:56 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:13:56 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:14:01	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-z4q42	Killing	Stopping container kube-rbac-proxy
22:14:01	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-z4q42	Killing	Stopping container kserve-container
22:14:04 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-z4q42	Unhealthy	Readiness probe failed: dial tcp 10.133.0.40:8080: connect: connection refused
22:14:04	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-5f849d69df-z4q42	Unhealthy	Readiness probe failed: Get "https://10.133.0.40:8643/healthz": dial tcp 10.133.0.40:8643: connect: connection refused
22:14:06	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-64tnq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:14:06	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-predictor-695f978978-64tnq	AddedInterface	Add eth0 [10.133.0.41/23] from ovn-kubernetes
22:14:06	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost": the object has been modified; please apply your changes to the latest version and try again
22:14:06 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost": the object has been modified; please apply your changes to the latest version and try again
22:14:06	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-predictor-695f978978	SuccessfulCreate	Created pod: isvc-predictive-xgboost-predictor-695f978978-64tnq
22:14:06	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-64tnq	Started	Started container storage-initializer
22:14:06	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-64tnq	Created	Created container: storage-initializer
22:14:06	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-predictor-695f978978 from 0 to 1
22:14:11	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-64tnq	Started	Started container kserve-container
22:14:11	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-64tnq	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:14:11	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-64tnq	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
22:14:11	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-64tnq	Created	Created container: kserve-container
22:14:11	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-64tnq	Created	Created container: kube-rbac-proxy
22:14:11	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-64tnq	Started	Started container kube-rbac-proxy
22:14:51 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:14:51 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:15:19 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:15:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	InferenceServiceReady	InferenceService [isvc-predictive-xgboost] is Ready
22:15:21 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:15:21 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:15:26	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-64tnq	Killing	Stopping container kube-rbac-proxy
22:15:26	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-64tnq	Killing	Stopping container kserve-container
22:15:29	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-64tnq	Unhealthy	Readiness probe failed: Get "https://10.133.0.41:8643/healthz": dial tcp 10.133.0.41:8643: connect: connection refused
22:15:29 (x9)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-695f978978-64tnq	Unhealthy	Readiness probe failed: dial tcp 10.133.0.41:8080: connect: connection refused
22:15:31	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-predictor-ff56bcbb5 from 0 to 1
22:15:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm": the object has been modified; please apply your changes to the latest version and try again
22:15:31	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-predictor-ff56bcbb5	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz
22:15:32	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz	AddedInterface	Add eth0 [10.133.0.42/23] from ovn-kubernetes
22:15:32	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:15:32	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz	Created	Created container: storage-initializer
22:15:32	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz	Started	Started container storage-initializer
22:15:36	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz	Started	Started container kserve-container
22:15:36	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz	Created	Created container: kserve-container
22:15:36	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz	Started	Started container kube-rbac-proxy
22:15:36	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz	Created	Created container: kube-rbac-proxy
22:15:36	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:15:36	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
22:15:46	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:15:46	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:16:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm] is Ready
22:16:54 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:17:01 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:17:01	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz	Killing	Stopping container kserve-container
22:17:01 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:17:01	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz	Killing	Stopping container kube-rbac-proxy
22:17:04 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz	Unhealthy	Readiness probe failed: dial tcp 10.133.0.42:8080: connect: connection refused
22:17:04	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz	Unhealthy	Readiness probe failed: Get "https://10.133.0.42:8643/healthz": dial tcp 10.133.0.42:8643: connect: connection refused
22:17:06	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-v2-predictor-745f9c8c9f from 0 to 1
22:17:07	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
22:17:07 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
22:17:07	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:17:07	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-v2-predictor-745f9c8c9f	SuccessfulCreate	Created pod: isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg
22:17:07	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg	Started	Started container storage-initializer
22:17:07	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg	Created	Created container: storage-initializer
22:17:07	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg	AddedInterface	Add eth0 [10.134.0.46/23] from ovn-kubernetes
22:17:12	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg	Pulling	Pulling image "kserve/predictiveserver:latest"
22:17:33	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg	Started	Started container kserve-container
22:17:33	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg	Created	Created container: kserve-container
22:17:33	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg	Started	Started container kube-rbac-proxy
22:17:33	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg	Created	Created container: kube-rbac-proxy
22:17:33	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg	Pulled	Successfully pulled image "kserve/predictiveserver:latest" in 20.83s (20.83s including waiting). Image size: 2353326736 bytes.
22:17:33	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:17:51 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:17:51 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:18:51	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg	Unhealthy	Readiness probe failed: Get "http://10.134.0.46:8080/v2/models/isvc-predictive-sklearn-v2/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
22:19:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	InferenceServiceReady	InferenceService [isvc-predictive-sklearn-v2] is Ready
22:19:00 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:19:07 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:19:07 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:19:07	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg	Killing	Stopping container kserve-container
22:19:07	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg	Killing	Stopping container kube-rbac-proxy
22:19:10	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg	Unhealthy	Readiness probe failed: Get "https://10.134.0.46:8643/healthz": dial tcp 10.134.0.46:8643: connect: connection refused
22:19:10 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg	Unhealthy	Readiness probe failed: Get "http://10.134.0.46:8080/v2/models/isvc-predictive-sklearn-v2/ready": dial tcp 10.134.0.46:8080: connect: connection refused
22:19:12	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-v2-predictor-868ff5dd6d	SuccessfulCreate	Created pod: isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c
22:19:12	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
22:19:12 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
22:19:12	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-v2-predictor-868ff5dd6d from 0 to 1
22:19:12	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
22:19:13	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c	AddedInterface	Add eth0 [10.133.0.43/23] from ovn-kubernetes
22:19:13	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c	Started	Started container storage-initializer
22:19:13	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:19:13	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c	Created	Created container: storage-initializer
22:19:17	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c	Created	Created container: kserve-container
22:19:17	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
22:19:17	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c	Started	Started container kserve-container
22:19:17	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:19:17	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c	Created	Created container: kube-rbac-proxy
22:19:17	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c	Started	Started container kube-rbac-proxy
22:19:42 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:19:42 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:20:34 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:20:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	InferenceServiceReady	InferenceService [isvc-predictive-xgboost-v2] is Ready
22:20:42	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c	Killing	Stopping container kserve-container
22:20:42	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c	Killing	Stopping container kube-rbac-proxy
22:20:42 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:20:42 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:20:44 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c	Unhealthy	Readiness probe failed: Get "http://10.133.0.43:8080/v2/models/isvc-predictive-xgboost-v2/ready": dial tcp 10.133.0.43:8080: connect: connection refused
22:20:44	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c	Unhealthy	Readiness probe failed: Get "https://10.133.0.43:8643/healthz": dial tcp 10.133.0.43:8643: connect: connection refused
22:20:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm-v2": the object has been modified; please apply your changes to the latest version and try again
22:20:47	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-v2-predictor-545b7cc587	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq
22:20:47	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-v2-predictor-545b7cc587 from 0 to 1
22:20:48	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:20:48	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq	Created	Created container: storage-initializer
22:20:48	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq	Started	Started container storage-initializer
22:20:48	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq	AddedInterface	Add eth0 [10.133.0.44/23] from ovn-kubernetes
22:20:52	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq	Created	Created container: kserve-container
22:20:52	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
22:20:53	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq	Started	Started container kube-rbac-proxy
22:20:53	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq	Started	Started container kserve-container
22:20:53	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:20:53	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq	Created	Created container: kube-rbac-proxy
22:21:17 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:21:17 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:22:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm-v2] is Ready
22:22:10 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:22:17	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq	Killing	Stopping container kube-rbac-proxy
22:22:17	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq	Killing	Stopping container kserve-container
22:22:17 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:22:17 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:22:19 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq	Unhealthy	Readiness probe failed: Get "http://10.133.0.44:8080/v2/models/isvc-predictive-lightgbm-v2/ready": dial tcp 10.133.0.44:8080: connect: connection refused
22:22:19	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq	Unhealthy	Readiness probe failed: Get "https://10.133.0.44:8643/healthz": dial tcp 10.133.0.44:8643: connect: connection refused
22:22:22	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-scheduler-predictor-699668459f	SuccessfulCreate	Created pod: isvc-sklearn-scheduler-predictor-699668459f-bmxrk
22:22:22 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-scheduler": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
22:22:22 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
22:22:22	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-scheduler-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-scheduler-predictor-699668459f from 0 to 1
22:22:23 (x6)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:22:30	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6c799687d7-2qq4t	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-predictor-serving-cert" not found
22:22:30	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn": the object has been modified; please apply your changes to the latest version and try again
22:22:30	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-predictor-6c799687d7 from 0 to 1
22:22:30	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-predictor-6c799687d7	SuccessfulCreate	Created pod: isvc-sklearn-predictor-6c799687d7-2qq4t
22:22:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6c799687d7-2qq4t	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:22:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6c799687d7-2qq4t	Created	Created container: storage-initializer
22:22:31	kserve-ci-e2e-test	multus	isvc-sklearn-predictor-6c799687d7-2qq4t	AddedInterface	Add eth0 [10.133.0.45/23] from ovn-kubernetes
22:22:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6c799687d7-2qq4t	Started	Started container storage-initializer
22:22:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6c799687d7-2qq4t	Created	Created container: kserve-container
22:22:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6c799687d7-2qq4t	Started	Started container kube-rbac-proxy
22:22:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6c799687d7-2qq4t	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine
22:22:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6c799687d7-2qq4t	Started	Started container kserve-container
22:22:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6c799687d7-2qq4t	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:22:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6c799687d7-2qq4t	Created	Created container: kube-rbac-proxy
22:23:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:23:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:23:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	InferenceServiceReady	InferenceService [isvc-sklearn] is Ready
22:23:53 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:24:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6c799687d7-2qq4t	Killing	Stopping container kube-rbac-proxy
22:24:00 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:24:00 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:24:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6c799687d7-2qq4t	Killing	Stopping container kserve-container
22:24:03 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6c799687d7-2qq4t	Unhealthy	Readiness probe failed: dial tcp 10.133.0.45:8080: connect: connection refused
22:24:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6c799687d7-2qq4t	Unhealthy	Readiness probe failed: Get "https://10.133.0.45:8643/healthz": dial tcp 10.133.0.45:8643: connect: connection refused
22:24:05	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4	Created	Created container: storage-initializer
22:24:05	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "sklearn-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
22:24:05 (x2)	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "sklearn-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "sklearn-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
22:24:05	kserve-ci-e2e-test	multus	sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4	AddedInterface	Add eth0 [10.133.0.46/23] from ovn-kubernetes
22:24:05	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:24:05	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4	Started	Started container storage-initializer
22:24:05	kserve-ci-e2e-test	replicaset-controller	sklearn-v2-mlserver-predictor-7f568cdc6b	SuccessfulCreate	Created pod: sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4
22:24:05	kserve-ci-e2e-test	deployment-controller	sklearn-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set sklearn-v2-mlserver-predictor-7f568cdc6b from 0 to 1
22:24:09	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
22:24:09	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4	Created	Created container: kserve-container
22:24:09	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4	Started	Started container kserve-container
22:24:09	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:24:09	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4	Created	Created container: kube-rbac-proxy
22:24:09	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4	Started	Started container kube-rbac-proxy
22:24:46	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
22:24:56	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	InferenceServiceReady	InferenceService [sklearn-v2-mlserver] is Ready
22:24:56 (x12)	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:25:05	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4	Killing	Stopping container kube-rbac-proxy
22:25:05 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:25:05	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4	Killing	Stopping container kserve-container
22:25:05 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:25:07	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4	Unhealthy	Readiness probe failed: Get "http://10.133.0.46:8080/v2/models/sklearn-v2-mlserver/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
22:25:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-runtime": the object has been modified; please apply your changes to the latest version and try again
22:25:10	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-runtime-predictor-cf66df4c5 from 0 to 1
22:25:10	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-runtime-predictor-cf66df4c5	SuccessfulCreate	Created pod: isvc-sklearn-runtime-predictor-cf66df4c5-64kkx
22:25:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-cf66df4c5-64kkx	Started	Started container storage-initializer
22:25:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-cf66df4c5-64kkx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:25:11 (x2)	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4	Unhealthy	Readiness probe failed: Get "https://10.133.0.46:8643/healthz": dial tcp 10.133.0.46:8643: connect: connection refused
22:25:11	kserve-ci-e2e-test	multus	isvc-sklearn-runtime-predictor-cf66df4c5-64kkx	AddedInterface	Add eth0 [10.134.0.47/23] from ovn-kubernetes
22:25:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-cf66df4c5-64kkx	Created	Created container: storage-initializer
22:25:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-cf66df4c5-64kkx	Created	Created container: kube-rbac-proxy
22:25:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-cf66df4c5-64kkx	Started	Started container kube-rbac-proxy
22:25:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-cf66df4c5-64kkx	Created	Created container: kserve-container
22:25:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-cf66df4c5-64kkx	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:25:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-cf66df4c5-64kkx	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine
22:25:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-cf66df4c5-64kkx	Started	Started container kserve-container
22:25:24 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-cf66df4c5-64kkx	Unhealthy	Readiness probe failed: dial tcp 10.134.0.47:8080: connect: connection refused
22:25:25	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:25:25	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:25:34 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:25:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-runtime] is Ready
22:25:40	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:25:40	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:25:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-cf66df4c5-64kkx	Killing	Stopping container kserve-container
22:25:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-cf66df4c5-64kkx	Killing	Stopping container kube-rbac-proxy
22:25:47	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-runtime-predictor-5f69c7d498 from 0 to 1
22:25:47	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-runtime-predictor-5f69c7d498	SuccessfulCreate	Created pod: isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6
22:25:47 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
22:25:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
22:25:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:25:48	kserve-ci-e2e-test	multus	isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6	AddedInterface	Add eth0 [10.133.0.47/23] from ovn-kubernetes
22:25:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6	Started	Started container storage-initializer
22:25:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6	Created	Created container: storage-initializer
22:25:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6	Started	Started container kube-rbac-proxy
22:25:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6	Created	Created container: kube-rbac-proxy
22:25:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:25:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6	Started	Started container kserve-container
22:25:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6	Created	Created container: kserve-container
22:25:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
22:26:02	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:26:02	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:26:30	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
22:26:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-v2-runtime] is Ready
22:26:40 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:26:47 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:26:47 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:26:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6	Killing	Stopping container kube-rbac-proxy
22:26:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6	Killing	Stopping container kserve-container
22:26:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6	Unhealthy	Readiness probe failed: Get "https://10.133.0.47:8643/healthz": dial tcp 10.133.0.47:8643: connect: connection refused
22:26:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6	Unhealthy	Readiness probe failed: Get "http://10.133.0.47:8080/v2/models/isvc-sklearn-v2-runtime/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
22:26:53	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-predictor-59d6dcff47 from 0 to 1
22:26:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-59d6dcff47-8nhk6	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-v2-predictor-serving-cert" not found
22:26:53	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-predictor-59d6dcff47	SuccessfulCreate	Created pod: isvc-sklearn-v2-predictor-59d6dcff47-8nhk6
22:26:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
22:26:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-59d6dcff47-8nhk6	Started	Started container storage-initializer
22:26:54	kserve-ci-e2e-test	multus	isvc-sklearn-v2-predictor-59d6dcff47-8nhk6	AddedInterface	Add eth0 [10.133.0.48/23] from ovn-kubernetes
22:26:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-59d6dcff47-8nhk6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:26:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-59d6dcff47-8nhk6	Created	Created container: storage-initializer
22:26:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-59d6dcff47-8nhk6	Created	Created container: kube-rbac-proxy
22:26:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-59d6dcff47-8nhk6	Created	Created container: kserve-container
22:26:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-59d6dcff47-8nhk6	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine
22:26:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-59d6dcff47-8nhk6	Started	Started container kserve-container
22:26:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-59d6dcff47-8nhk6	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:26:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-59d6dcff47-8nhk6	Started	Started container kube-rbac-proxy
22:27:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:27:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:28:16 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:28:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	InferenceServiceReady	InferenceService [isvc-sklearn-v2] is Ready
22:28:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-59d6dcff47-8nhk6	Killing	Stopping container kserve-container
22:28:23 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:28:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-59d6dcff47-8nhk6	Killing	Stopping container kube-rbac-proxy
22:28:23 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:28:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-59d6dcff47-8nhk6	Unhealthy	Readiness probe failed: Get "https://10.133.0.48:8643/healthz": dial tcp 10.133.0.48:8643: connect: connection refused
22:28:26 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-59d6dcff47-8nhk6	Unhealthy	Readiness probe failed: dial tcp 10.133.0.48:8080: connect: connection refused
22:28:28	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-mixed-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-mixed-predictor-7ddbdd7546 from 0 to 1
22:28:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
22:28:28	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-mixed-predictor-7ddbdd7546	SuccessfulCreate	Created pod: isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d
22:28:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-mixed": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-mixed": the object has been modified; please apply your changes to the latest version and try again
22:28:29	kserve-ci-e2e-test	multus	isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d	AddedInterface	Add eth0 [10.133.0.49/23] from ovn-kubernetes
22:28:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:28:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d	Created	Created container: storage-initializer
22:28:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d	Started	Started container storage-initializer
22:28:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d	Started	Started container kube-rbac-proxy
22:28:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine
22:28:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d	Created	Created container: kube-rbac-proxy
22:28:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:28:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d	Started	Started container kserve-container
22:28:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d	Created	Created container: kserve-container
22:29:13 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:29:13 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:29:51	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	InferenceServiceReady	InferenceService [isvc-sklearn-v2-mixed] is Ready
22:29:51 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:29:58 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:29:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d	Killing	Stopping container kserve-container
22:29:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d	Killing	Stopping container kube-rbac-proxy
22:29:58 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:30:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d	Unhealthy	Readiness probe failed: Get "https://10.133.0.49:8643/healthz": dial tcp 10.133.0.49:8643: connect: connection refused
22:30:01 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d	Unhealthy	Readiness probe failed: dial tcp 10.133.0.49:8080: connect: connection refused
22:30:03	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-predictor-547dfc5669	SuccessfulCreate	Created pod: isvc-tensorflow-predictor-547dfc5669-hnppd
22:30:03 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow": the object has been modified; please apply your changes to the latest version and try again
22:30:03	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-predictor-547dfc5669 from 0 to 1
22:30:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow": the object has been modified; please apply your changes to the latest version and try again
22:30:04	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-hnppd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:30:04	kserve-ci-e2e-test	multus	isvc-tensorflow-predictor-547dfc5669-hnppd	AddedInterface	Add eth0 [10.134.0.48/23] from ovn-kubernetes
22:30:04	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-hnppd	Created	Created container: storage-initializer
22:30:04	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-hnppd	Started	Started container storage-initializer
22:30:09	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-hnppd	Pulling	Pulling image "tensorflow/serving:2.6.2"
22:30:13	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-hnppd	Created	Created container: kserve-container
22:30:13	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-hnppd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:30:13	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-hnppd	Started	Started container kube-rbac-proxy
22:30:13	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-hnppd	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 3.722s (3.722s including waiting). Image size: 425873876 bytes.
22:30:13	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-hnppd	Created	Created container: kube-rbac-proxy
22:30:13	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-hnppd	Started	Started container kserve-container
22:30:19 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-hnppd	Unhealthy	Readiness probe failed: dial tcp 10.134.0.48:8080: connect: connection refused
22:30:29	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	InferenceServiceReady	InferenceService [isvc-tensorflow] is Ready
22:30:29 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:30:33 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:30:33 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:30:44	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-hnppd	Killing	Stopping container kserve-container
22:30:44	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-hnppd	Killing	Stopping container kube-rbac-proxy
22:30:49	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-runtime-predictor-855cfcd677 from 0 to 1
22:30:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow-runtime": the object has been modified; please apply your changes to the latest version and try again
22:30:50	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-runtime-predictor-855cfcd677	SuccessfulCreate	Created pod: isvc-tensorflow-runtime-predictor-855cfcd677-7699m
22:30:50	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-7699m	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
22:30:51	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-7699m	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:30:51	kserve-ci-e2e-test	multus	isvc-tensorflow-runtime-predictor-855cfcd677-7699m	AddedInterface	Add eth0 [10.133.0.50/23] from ovn-kubernetes
22:30:51	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-7699m	Created	Created container: storage-initializer
22:30:51	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-7699m	Started	Started container storage-initializer
22:30:56	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-7699m	Pulling	Pulling image "tensorflow/serving:2.6.2"
22:30:59	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-7699m	Created	Created container: kube-rbac-proxy
22:30:59	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-7699m	Started	Started container kube-rbac-proxy
22:30:59	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-7699m	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 3.64s (3.64s including waiting). Image size: 425873876 bytes.
22:30:59	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-7699m	Created	Created container: kserve-container
22:30:59	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-7699m	Started	Started container kserve-container
22:30:59	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-7699m	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:31:06 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-7699m	Unhealthy	Readiness probe failed: dial tcp 10.133.0.50:8080: connect: connection refused
22:31:14 (x6)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-547dfc5669-hnppd	Unhealthy	Readiness probe failed: Get "https://10.134.0.48:8643/healthz": dial tcp 10.134.0.48:8643: connect: connection refused
22:31:16 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:31:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	InferenceServiceReady	InferenceService [isvc-tensorflow-runtime] is Ready
22:31:20 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:31:20 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:31:30	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-7699m	Killing	Stopping container kube-rbac-proxy
22:31:30	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-7699m	Killing	Stopping container kserve-container
22:31:35	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-cztrf	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:31:35	kserve-ci-e2e-test	replicaset-controller	isvc-triton-predictor-5d7db58676	SuccessfulCreate	Created pod: isvc-triton-predictor-5d7db58676-cztrf
22:31:35	kserve-ci-e2e-test	multus	isvc-triton-predictor-5d7db58676-cztrf	AddedInterface	Add eth0 [10.134.0.49/23] from ovn-kubernetes
22:31:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	UpdateFailed	Failed to update status for InferenceService "isvc-triton": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-triton": the object has been modified; please apply your changes to the latest version and try again
22:31:35	kserve-ci-e2e-test	deployment-controller	isvc-triton-predictor	ScalingReplicaSet	Scaled up replica set isvc-triton-predictor-5d7db58676 from 0 to 1
22:31:36	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-cztrf	Created	Created container: storage-initializer
22:31:36	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-cztrf	Started	Started container storage-initializer
22:31:40	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-cztrf	Pulling	Pulling image "nvcr.io/nvidia/tritonserver:23.05-py3"
22:31:56 (x6)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-855cfcd677-7699m	Unhealthy	Readiness probe failed: Get "https://10.133.0.50:8643/healthz": dial tcp 10.133.0.50:8643: connect: connection refused
22:33:35	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-cztrf	Pulled	Successfully pulled image "nvcr.io/nvidia/tritonserver:23.05-py3" in 1m55.567s (1m55.567s including waiting). Image size: 12907074623 bytes.
22:33:35	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-cztrf	Created	Created container: kserve-container
22:33:35	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-cztrf	Started	Started container kserve-container
22:33:35	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-cztrf	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:33:35	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-cztrf	Created	Created container: kube-rbac-proxy
22:33:35	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-cztrf	Started	Started container kube-rbac-proxy
22:33:35 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:33:35 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:33:38 (x2)	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-cztrf	Unhealthy	Readiness probe failed: dial tcp 10.134.0.49:8080: connect: connection refused
22:33:43 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:33:43	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	InferenceServiceReady	InferenceService [isvc-triton] is Ready
22:33:47	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-cztrf	Killing	Stopping container kube-rbac-proxy
22:33:47	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-cztrf	Killing	Stopping container kserve-container
22:33:48	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-5d7db58676-cztrf	Unhealthy	Readiness probe failed: Get "https://10.134.0.49:8643/healthz": dial tcp 10.134.0.49:8643: connect: connection refused
22:33:52	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-predictor-7566d65bf7 from 0 to 1
22:33:53	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-tc894	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-predictor-serving-cert" not found
22:33:53	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-predictor-7566d65bf7	SuccessfulCreate	Created pod: isvc-xgboost-predictor-7566d65bf7-tc894
22:33:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost": the object has been modified; please apply your changes to the latest version and try again
22:33:54	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-tc894	Created	Created container: storage-initializer
22:33:54	kserve-ci-e2e-test	multus	isvc-xgboost-predictor-7566d65bf7-tc894	AddedInterface	Add eth0 [10.133.0.51/23] from ovn-kubernetes
22:33:54	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-tc894	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:33:54	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-tc894	Started	Started container storage-initializer
22:33:58	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-tc894	Pulling	Pulling image "kserve/xgbserver:latest"
22:34:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-tc894	Started	Started container kube-rbac-proxy
22:34:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-tc894	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 17.279s (17.279s including waiting). Image size: 1307355899 bytes.
22:34:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-tc894	Created	Created container: kserve-container
22:34:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-tc894	Started	Started container kserve-container
22:34:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-tc894	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:34:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-tc894	Created	Created container: kube-rbac-proxy
22:34:53 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:34:53 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:35:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	InferenceServiceReady	InferenceService [isvc-xgboost] is Ready
22:35:14 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:35:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:35:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:35:23 (x8)	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-tc894	Unhealthy	Readiness probe failed: dial tcp 10.133.0.51:8080: connect: connection refused
22:35:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-tc894	Killing	Stopping container kserve-container
22:35:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-tc894	Killing	Stopping container kube-rbac-proxy
22:35:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-7566d65bf7-tc894	Unhealthy	Readiness probe failed: Get "https://10.133.0.51:8643/healthz": dial tcp 10.133.0.51:8643: connect: connection refused
22:35:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
22:35:28	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
22:35:28	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-mlserver-predictor-7b55b84f54 from 0 to 1
22:35:28 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
22:35:28	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-mlserver-predictor-7b55b84f54	SuccessfulCreate	Created pod: isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh
22:35:29	kserve-ci-e2e-test	multus	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh	AddedInterface	Add eth0 [10.133.0.52/23] from ovn-kubernetes
22:35:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:35:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh	Created	Created container: storage-initializer
22:35:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh	Started	Started container storage-initializer
22:35:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh	Started	Started container kserve-container
22:35:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh	Created	Created container: kube-rbac-proxy
22:35:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh	Started	Started container kube-rbac-proxy
22:35:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:35:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
22:35:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh	Created	Created container: kserve-container
22:35:43	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:35:43	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:36:12 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:36:12	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	InferenceServiceReady	InferenceService [isvc-xgboost-v2-mlserver] is Ready
22:36:13 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:36:13 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:36:18	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh	Killing	Stopping container kserve-container
22:36:18	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh	Killing	Stopping container kube-rbac-proxy
22:36:22	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh	Unhealthy	Readiness probe failed: Get "https://10.133.0.52:8643/healthz": dial tcp 10.133.0.52:8643: connect: connection refused
22:36:22	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh	Unhealthy	Readiness probe failed: Get "http://10.133.0.52:8080/v2/models/isvc-xgboost-v2-mlserver/ready": dial tcp 10.133.0.52:8080: connect: connection refused
22:36:23	kserve-ci-e2e-test	replicaset-controller	xgboost-v2-mlserver-predictor-59944bcf4c	SuccessfulCreate	Created pod: xgboost-v2-mlserver-predictor-59944bcf4c-pztsc
22:36:23	kserve-ci-e2e-test	deployment-controller	xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set xgboost-v2-mlserver-predictor-59944bcf4c from 0 to 1
22:36:23	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
22:36:24	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-pztsc	Created	Created container: storage-initializer
22:36:24	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-pztsc	Started	Started container storage-initializer
22:36:24	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-pztsc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:36:24	kserve-ci-e2e-test	multus	xgboost-v2-mlserver-predictor-59944bcf4c-pztsc	AddedInterface	Add eth0 [10.133.0.53/23] from ovn-kubernetes
22:36:28	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-pztsc	Created	Created container: kserve-container
22:36:28	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-pztsc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:36:28	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-pztsc	Created	Created container: kube-rbac-proxy
22:36:28	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-pztsc	Started	Started container kube-rbac-proxy
22:36:28	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-pztsc	Started	Started container kserve-container
22:36:28	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-pztsc	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
22:37:06 (x10)	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:37:06	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	InferenceServiceReady	InferenceService [xgboost-v2-mlserver] is Ready
22:37:08 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:37:08 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:37:13	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-pztsc	Killing	Stopping container kube-rbac-proxy
22:37:13	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-pztsc	Killing	Stopping container kserve-container
22:37:16	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-pztsc	Unhealthy	Readiness probe failed: Get "http://10.133.0.53:8080/v2/models/xgboost-v2-mlserver/ready": dial tcp 10.133.0.53:8080: connect: connection refused
22:37:16	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-59944bcf4c-pztsc	Unhealthy	Readiness probe failed: Get "https://10.133.0.53:8643/healthz": dial tcp 10.133.0.53:8643: connect: connection refused
22:37:18	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-runtime-predictor-54dd488c85	SuccessfulCreate	Created pod: isvc-xgboost-runtime-predictor-54dd488c85-bwgjp
22:37:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-runtime": the object has been modified; please apply your changes to the latest version and try again
22:37:18	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-runtime-predictor-54dd488c85 from 0 to 1
22:37:19	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-bwgjp	Created	Created container: storage-initializer
22:37:19	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-bwgjp	Started	Started container storage-initializer
22:37:19	kserve-ci-e2e-test	multus	isvc-xgboost-runtime-predictor-54dd488c85-bwgjp	AddedInterface	Add eth0 [10.133.0.54/23] from ovn-kubernetes
22:37:19	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-bwgjp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:37:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-bwgjp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:37:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-bwgjp	Pulled	Container image "kserve/xgbserver:latest" already present on machine
22:37:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-bwgjp	Created	Created container: kube-rbac-proxy
22:37:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-bwgjp	Started	Started container kube-rbac-proxy
22:37:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-bwgjp	Created	Created container: kserve-container
22:37:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-bwgjp	Started	Started container kserve-container
22:38:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:38:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:38:18 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:38:18 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:38:21	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-runtime] is Ready
22:38:21 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:38:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-bwgjp	Killing	Stopping container kserve-container
22:38:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-bwgjp	Killing	Stopping container kube-rbac-proxy
22:38:31 (x8)	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-bwgjp	Unhealthy	Readiness probe failed: dial tcp 10.133.0.54:8080: connect: connection refused
22:38:31	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-54dd488c85-bwgjp	Unhealthy	Readiness probe failed: Get "https://10.133.0.54:8643/healthz": dial tcp 10.133.0.54:8643: connect: connection refused
22:38:34	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-runtime-predictor-7787c55dc9	SuccessfulCreate	Created pod: isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp
22:38:34	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-runtime-predictor-7787c55dc9 from 0 to 1
22:38:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
22:38:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp	Started	Started container storage-initializer
22:38:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp	Created	Created container: storage-initializer
22:38:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:38:34	kserve-ci-e2e-test	multus	isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp	AddedInterface	Add eth0 [10.133.0.55/23] from ovn-kubernetes
22:38:38	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
22:38:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp	Started	Started container kube-rbac-proxy
22:38:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp	Created	Created container: kube-rbac-proxy
22:38:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp	Created	Created container: kserve-container
22:38:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp	Started	Started container kserve-container
22:38:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:38:49	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:38:49	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:39:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
22:39:27 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:39:27	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-v2-runtime] is Ready
22:39:34 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:39:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp	Killing	Stopping container kserve-container
22:39:34 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:39:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp	Killing	Stopping container kube-rbac-proxy
22:39:36	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp	Unhealthy	Readiness probe failed: Get "https://10.133.0.55:8643/healthz": dial tcp 10.133.0.55:8643: connect: connection refused
22:39:39	kserve-ci-e2e-test	multus	isvc-xgboost-v2-predictor-fcc668bc-m76lz	AddedInterface	Add eth0 [10.133.0.56/23] from ovn-kubernetes
22:39:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-m76lz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:39:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
22:39:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-m76lz	Created	Created container: storage-initializer
22:39:39	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-predictor-fcc668bc	SuccessfulCreate	Created pod: isvc-xgboost-v2-predictor-fcc668bc-m76lz
22:39:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-m76lz	Started	Started container storage-initializer
22:39:39	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-predictor-fcc668bc from 0 to 1
22:39:44	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-m76lz	Pulled	Container image "kserve/xgbserver:latest" already present on machine
22:39:44	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-m76lz	Started	Started container kserve-container
22:39:44	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-m76lz	Created	Created container: kserve-container
22:39:44	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-m76lz	Started	Started container kube-rbac-proxy
22:39:44	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-m76lz	Created	Created container: kube-rbac-proxy
22:39:44	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-m76lz	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:40:09 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:40:09 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:40:39 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:40:39 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:40:42 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:40:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	InferenceServiceReady	InferenceService [isvc-xgboost-v2] is Ready
22:40:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-m76lz	Killing	Stopping container kube-rbac-proxy
22:40:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-m76lz	Killing	Stopping container kserve-container
22:40:52 (x8)	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-m76lz	Unhealthy	Readiness probe failed: dial tcp 10.133.0.56:8080: connect: connection refused
22:40:52	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-fcc668bc-m76lz	Unhealthy	Readiness probe failed: Get "https://10.133.0.56:8643/healthz": dial tcp 10.133.0.56:8643: connect: connection refused
22:40:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3": the object has been modified; please apply your changes to the latest version and try again
22:40:54	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-predictor-7d6947b598	SuccessfulCreate	Created pod: isvc-sklearn-s3-predictor-7d6947b598-66vn6
22:40:54	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-predictor-7d6947b598 from 0 to 1
22:40:55	kserve-ci-e2e-test	multus	isvc-sklearn-s3-predictor-7d6947b598-66vn6	AddedInterface	Add eth0 [10.133.0.57/23] from ovn-kubernetes
22:40:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-7d6947b598-66vn6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:40:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-7d6947b598-66vn6	Created	Created container: storage-initializer
22:40:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-7d6947b598-66vn6	Started	Started container storage-initializer
22:40:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-7d6947b598-66vn6	Started	Started container kserve-container
22:40:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-7d6947b598-66vn6	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine
22:40:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-7d6947b598-66vn6	Created	Created container: kserve-container
22:40:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-7d6947b598-66vn6	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:40:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-7d6947b598-66vn6	Created	Created container: kube-rbac-proxy
22:40:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-7d6947b598-66vn6	Started	Started container kube-rbac-proxy
22:42:08 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:42:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	InferenceServiceReady	InferenceService [isvc-sklearn-s3] is Ready
22:42:09 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:42:09 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:42:14	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d from 0 to 1
22:42:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-7d6947b598-66vn6	Killing	Stopping container kube-rbac-proxy
22:42:14	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8
22:42:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-pass": the object has been modified; please apply your changes to the latest version and try again
22:42:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-7d6947b598-66vn6	Killing	Stopping container kserve-container
22:42:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8	Created	Created container: storage-initializer
22:42:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8	Started	Started container storage-initializer
22:42:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:42:15	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8	AddedInterface	Add eth0 [10.134.0.50/23] from ovn-kubernetes
22:42:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8	Started	Started container kserve-container
22:42:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine
22:42:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8	Created	Created container: kserve-container
22:42:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8	Started	Started container kube-rbac-proxy
22:42:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8	Created	Created container: kube-rbac-proxy
22:42:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:42:18 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-7d6947b598-66vn6	Unhealthy	Readiness probe failed: dial tcp 10.133.0.57:8080: connect: connection refused
22:42:44 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:42:44 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:43:25 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8	Unhealthy	Readiness probe failed: dial tcp 10.134.0.50:8080: connect: connection refused
22:43:30 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:43:30 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:43:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-global-pass] is Ready
22:43:35 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:43:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8	Killing	Stopping container kube-rbac-proxy
22:43:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8	Killing	Stopping container kserve-container
22:43:46 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-fail": the object has been modified; please apply your changes to the latest version and try again
22:43:46	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-fail": the object has been modified; please apply your changes to the latest version and try again
22:43:46	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894 from 0 to 1
22:43:46	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t
22:43:47	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t	AddedInterface	Add eth0 [10.133.0.58/23] from ovn-kubernetes
22:43:51 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t	Started	Started container storage-initializer
22:43:51 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t	Created	Created container: storage-initializer
22:43:51 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:43:51 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:43:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t	BackOff	Back-off restarting failed container storage-initializer in pod isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t_kserve-ci-e2e-test(4561a3ef-c9cb-4155-8e56-9da51a8ef5fc)
22:43:56 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
22:43:57	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8 from 0 to 1
22:43:57 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-pass": the object has been modified; please apply your changes to the latest version and try again
22:43:57	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-pass": the object has been modified; please apply your changes to the latest version and try again
22:43:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
22:43:57	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8
22:43:58	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8	AddedInterface	Add eth0 [10.133.0.59/23] from ovn-kubernetes
22:43:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8	Started	Started container storage-initializer
22:43:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:43:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8	Created	Created container: storage-initializer
22:44:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine
22:44:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8	Started	Started container kube-rbac-proxy
22:44:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8	Created	Created container: kube-rbac-proxy
22:44:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:44:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8	Started	Started container kserve-container
22:44:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8	Created	Created container: kserve-container
22:44:27 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:44:27 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:45:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-custom-pass] is Ready
22:45:11 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:45:12 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:45:12 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:45:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8	Killing	Stopping container kserve-container
22:45:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8	Killing	Stopping container kube-rbac-proxy
22:45:18	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4 from 0 to 1
22:45:18	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj
22:45:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
22:45:18 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-fail": the object has been modified; please apply your changes to the latest version and try again
22:45:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-fail": the object has been modified; please apply your changes to the latest version and try again
22:45:19	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj	AddedInterface	Add eth0 [10.134.0.51/23] from ovn-kubernetes
22:45:21 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8	Unhealthy	Readiness probe failed: dial tcp 10.133.0.59:8080: connect: connection refused
22:45:22 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:45:22 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj	Started	Started container storage-initializer
22:45:22 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj	Created	Created container: storage-initializer
22:45:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj	BackOff	Back-off restarting failed container storage-initializer in pod isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj_kserve-ci-e2e-test(932c59b3-7e1f-43c6-92d8-22824f30a729)
22:45:28 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:45:29	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv
22:45:29	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c from 0 to 1
22:45:29 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-pass": the object has been modified; please apply your changes to the latest version and try again
22:45:29	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-pass": the object has been modified; please apply your changes to the latest version and try again
22:45:30 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
22:45:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv	Started	Started container storage-initializer
22:45:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv	Created	Created container: storage-initializer
22:45:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:45:31	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv	AddedInterface	Add eth0 [10.133.0.60/23] from ovn-kubernetes
22:45:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv	Created	Created container: kube-rbac-proxy
22:45:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine
22:45:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv	Started	Started container kube-rbac-proxy
22:45:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:45:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv	Started	Started container kserve-container
22:45:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv	Created	Created container: kserve-container
22:45:59 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:45:59 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:46:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-serving-pass] is Ready
22:46:50 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:46:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv	Killing	Stopping container kube-rbac-proxy
22:46:59 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:46:59 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:46:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv	Killing	Stopping container kserve-container
22:47:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-fail": the object has been modified; please apply your changes to the latest version and try again
22:47:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv	Unhealthy	Readiness probe failed: Get "https://10.133.0.60:8643/healthz": dial tcp 10.133.0.60:8643: connect: connection refused
22:47:00 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv	Unhealthy	Readiness probe failed: dial tcp 10.133.0.60:8080: connect: connection refused
22:47:00	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7 from 0 to 1
22:47:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
22:47:00	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q
22:47:01	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q	AddedInterface	Add eth0 [10.134.0.52/23] from ovn-kubernetes
22:47:05 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q	Created	Created container: storage-initializer
22:47:05 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine
22:47:05 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q	Started	Started container storage-initializer
22:47:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q	BackOff	Back-off restarting failed container storage-initializer in pod isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q_kserve-ci-e2e-test(e9d41004-86b4-44ce-8f05-ac8e0683d139)
22:47:09 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.