Events

Time	Namespace	Component	RelatedObject	Reason	Message
23:02:22	kserve-ci-e2e-test		isvc-tensorflow-predictor-6756f669d7-gwwz8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-gwwz8 to ip-10-0-133-183.ec2.internal
22:35:15	kserve-ci-e2e-test		isvc-sklearn-mcp-predictor-5fdf4889b4-t2fgn	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5fdf4889b4-t2fgn to ip-10-0-133-183.ec2.internal
22:38:48	kserve-ci-e2e-test		isvc-paddle-v2-kserve-predictor-7dbd59854-t4fqm	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-t4fqm to ip-10-0-133-183.ec2.internal
22:44:34	kserve-ci-e2e-test		isvc-primary-0d7654-predictor-ff757f45b-xphvm	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-primary-0d7654-predictor-ff757f45b-xphvm to ip-10-0-133-183.ec2.internal
22:36:35	kserve-ci-e2e-test		isvc-paddle-predictor-6b8b7cfb4b-6hx29	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-6hx29 to ip-10-0-133-183.ec2.internal
22:45:54	kserve-ci-e2e-test		isvc-secondary-0d7654-predictor-6647fcc567-bpj6r	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-secondary-0d7654-predictor-6647fcc567-bpj6r to ip-10-0-133-183.ec2.internal
22:24:44	kserve-ci-e2e-test		isvc-sklearn-batcher-custom-predictor-ccbd696dd-w2mzz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-ccbd696dd-w2mzz to ip-10-0-133-183.ec2.internal
22:34:25	kserve-ci-e2e-test		isvc-mlflow-v2-runtime-predictor-5fdb47d546-7l7dn	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-7l7dn to ip-10-0-133-183.ec2.internal
23:09:16	kserve-ci-e2e-test		isvc-xgboost-runtime-predictor-779db84d9-z2km9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-z2km9 to ip-10-0-133-183.ec2.internal
22:51:11	kserve-ci-e2e-test		isvc-predictive-sklearn-v2-predictor-b5d4f6b79-8ns7j	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-8ns7j to ip-10-0-133-183.ec2.internal
22:26:29	kserve-ci-e2e-test		isvc-logger-predictor-64d54fcc88-bsbws	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-logger-predictor-64d54fcc88-bsbws to ip-10-0-133-183.ec2.internal
22:37:47	kserve-ci-e2e-test		isvc-paddle-runtime-predictor-7f4d4f9dc8-m76wv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-m76wv to ip-10-0-133-183.ec2.internal
22:27:54	kserve-ci-e2e-test		isvc-lightgbm-predictor-bdf964bd-z5nft	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-z5nft to ip-10-0-133-183.ec2.internal
23:07:36	kserve-ci-e2e-test		isvc-xgboost-v2-mlserver-predictor-67d4bc6646-rrdgw	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-rrdgw to ip-10-0-133-183.ec2.internal
23:11:36	kserve-ci-e2e-test		isvc-xgboost-v2-predictor-6fcdd6977c-xctlh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-xctlh to ip-10-0-133-183.ec2.internal
23:10:36	kserve-ci-e2e-test		isvc-xgboost-v2-runtime-predictor-6dc5954dc-xhb2s	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-xhb2s to ip-10-0-133-183.ec2.internal
22:26:09	kserve-ci-e2e-test		message-dumper-predictor-c7d86bcbd-dqtxz	Scheduled	Successfully assigned kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-dqtxz to ip-10-0-133-183.ec2.internal
22:46:20	kserve-ci-e2e-test		isvc-predictive-sklearn-predictor-cd7c759c9-hqwrv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-hqwrv to ip-10-0-133-183.ec2.internal
22:23:08	kserve-ci-e2e-test		isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq to ip-10-0-133-183.ec2.internal
22:31:05	kserve-ci-e2e-test		isvc-lightgbm-v2-runtime-predictor-8765c9667-gmjt6	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-gmjt6 to ip-10-0-133-183.ec2.internal
22:41:32	kserve-ci-e2e-test		isvc-pmml-runtime-predictor-67bc544947-dkqsr	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-dkqsr to ip-10-0-133-183.ec2.internal
23:05:55	kserve-ci-e2e-test		isvc-xgboost-predictor-8689c4cfcc-67n8f	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-67n8f to ip-10-0-133-183.ec2.internal
23:03:44	kserve-ci-e2e-test		isvc-triton-predictor-84bb65d94b-nprnq	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-nprnq to ip-10-0-133-183.ec2.internal
23:03:03	kserve-ci-e2e-test		isvc-tensorflow-runtime-predictor-8699d78cf-6kldd	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-6kldd to ip-10-0-133-183.ec2.internal
22:58:41	kserve-ci-e2e-test		isvc-sklearn-v2-runtime-predictor-6d84c876f4-2r472	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-2r472 to ip-10-0-133-183.ec2.internal
22:46:10	kserve-ci-e2e-test		isvc-init-fail-241fac-predictor-79dffbcd5f-c5zcg	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-init-fail-241fac-predictor-79dffbcd5f-c5zcg to ip-10-0-133-183.ec2.internal
22:57:04	kserve-ci-e2e-test		sklearn-v2-mlserver-predictor-65d8664766-f4fdh	Scheduled	Successfully assigned kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-f4fdh to ip-10-0-133-183.ec2.internal
23:08:26	kserve-ci-e2e-test		xgboost-v2-mlserver-predictor-7799869d6f-5ftr9	Scheduled	Successfully assigned kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-5ftr9 to ip-10-0-133-183.ec2.internal
22:58:04	kserve-ci-e2e-test		isvc-sklearn-runtime-predictor-65cd49579f-dxsln	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-65cd49579f-dxsln to ip-10-0-133-183.ec2.internal
22:49:41	kserve-ci-e2e-test		isvc-predictive-lightgbm-predictor-75cb94f9f-hwz4l	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-hwz4l to ip-10-0-133-183.ec2.internal
22:52:41	kserve-ci-e2e-test		isvc-predictive-xgboost-v2-predictor-6577c65fd8-5sqbb	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-5sqbb to ip-10-0-133-183.ec2.internal
22:54:11	kserve-ci-e2e-test		isvc-predictive-lightgbm-v2-predictor-865b4598f7-t58bp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-t58bp to ip-10-0-133-183.ec2.internal
22:34:05	kserve-ci-e2e-test		isvc-lightgbm-v2-kserve-predictor-559bf6989-vdclq	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-vdclq to ip-10-0-133-183.ec2.internal
22:59:41	kserve-ci-e2e-test		isvc-sklearn-v2-predictor-69755fbb9-kh7dl	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-predictor-69755fbb9-kh7dl to ip-10-0-133-183.ec2.internal
23:01:02	kserve-ci-e2e-test		isvc-sklearn-v2-mixed-predictor-7f8b779bc6-82fxz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7f8b779bc6-82fxz to ip-10-0-133-183.ec2.internal
22:29:34	kserve-ci-e2e-test		isvc-lightgbm-runtime-predictor-749c4f6d58-28jjm	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-28jjm to ip-10-0-133-183.ec2.internal
23:17:21	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-n2hcp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-n2hcp to ip-10-0-133-183.ec2.internal
23:18:32	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-8r5vr	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-8r5vr to ip-10-0-133-183.ec2.internal
23:14:17	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-rkvjg	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-rkvjg to ip-10-0-133-183.ec2.internal
22:43:03	kserve-ci-e2e-test		isvc-pmml-v2-kserve-predictor-6578f8ffc7-vzp5s	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-vzp5s to ip-10-0-133-183.ec2.internal
23:15:38	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-7fb4l	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-7fb4l to ip-10-0-133-183.ec2.internal
23:15:49	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-nmm2l	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-nmm2l to ip-10-0-133-183.ec2.internal
23:17:10	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-pqvbj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-pqvbj to ip-10-0-133-183.ec2.internal
23:12:56	kserve-ci-e2e-test		isvc-sklearn-s3-predictor-88457d696-6fqtp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-predictor-88457d696-6fqtp to ip-10-0-133-183.ec2.internal
22:39:50	kserve-ci-e2e-test		isvc-pmml-predictor-8bb578669-p57jg	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-p57jg to ip-10-0-133-183.ec2.internal
22:55:44	kserve-ci-e2e-test		isvc-sklearn-predictor-d8dbfbbb9-j5mxm	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-predictor-d8dbfbbb9-j5mxm to ip-10-0-133-183.ec2.internal
22:48:11	kserve-ci-e2e-test		isvc-predictive-xgboost-predictor-7ff98fd74d-fhw8t	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-fhw8t to ip-10-0-133-183.ec2.internal
22:23:08	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-predictor-6c75bdff6f	SuccessfulCreate	Created pod: isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq
22:23:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
22:23:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
22:23:08	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-predictor-6c75bdff6f from 0 to 1
22:23:09	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq	AddedInterface	Add eth0 [10.133.0.28/23] from ovn-kubernetes
22:23:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
22:23:10	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b"
22:23:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" in 3.138s (3.138s including waiting). Image size: 299671465 bytes.
22:23:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq	Started	Started container storage-initializer
22:23:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq	Created	Created container: storage-initializer
22:23:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1293"
22:23:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq	Started	Started container kserve-container
22:23:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq	Created	Created container: kserve-container
22:23:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
22:23:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1293" in 12.034s (12.034s including waiting). Image size: 1560615339 bytes.
22:23:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq	Created	Created container: kube-rbac-proxy
22:23:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq	Pulling	Pulling image "quay.io/opendatahub/kserve-agent@sha256:cbf2f0964114aa89f1941320ac63f4a95a47d78bad8157d48396be0e3b5c023b"
22:23:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq	Started	Started container kube-rbac-proxy
22:23:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.537s (2.537s including waiting). Image size: 211946088 bytes.
22:23:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq	Started	Started container agent
22:23:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq	Created	Created container: agent
22:23:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:cbf2f0964114aa89f1941320ac63f4a95a47d78bad8157d48396be0e3b5c023b" in 2.227s (2.227s including waiting). Image size: 237897291 bytes.
22:24:09 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:24:09 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:24:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InferenceServiceReady	InferenceService [isvc-sklearn-batcher] is Ready
22:24:38 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:24:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:24:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:24:44	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-custom-predictor-ccbd696dd	SuccessfulCreate	Created pod: isvc-sklearn-batcher-custom-predictor-ccbd696dd-w2mzz
22:24:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher-custom": the object has been modified; please apply your changes to the latest version and try again
22:24:44	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-custom-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-custom-predictor-ccbd696dd from 0 to 1
22:24:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq	Killing	Stopping container kserve-container
22:24:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq	Killing	Stopping container agent
22:24:44 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher-custom": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher-custom": the object has been modified; please apply your changes to the latest version and try again
22:24:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-ccbd696dd-w2mzz	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
22:24:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq	Killing	Stopping container kube-rbac-proxy
22:24:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-ccbd696dd-w2mzz	Started	Started container storage-initializer
22:24:45	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-custom-predictor-ccbd696dd-w2mzz	AddedInterface	Add eth0 [10.133.0.29/23] from ovn-kubernetes
22:24:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-ccbd696dd-w2mzz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:24:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-ccbd696dd-w2mzz	Created	Created container: storage-initializer
22:24:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-ccbd696dd-w2mzz	Started	Started container kube-rbac-proxy
22:24:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-ccbd696dd-w2mzz	Created	Created container: kserve-container
22:24:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-ccbd696dd-w2mzz	Created	Created container: kube-rbac-proxy
22:24:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-ccbd696dd-w2mzz	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:cbf2f0964114aa89f1941320ac63f4a95a47d78bad8157d48396be0e3b5c023b" already present on machine
22:24:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-ccbd696dd-w2mzz	Started	Started container kserve-container
22:24:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-ccbd696dd-w2mzz	Started	Started container agent
22:24:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-ccbd696dd-w2mzz	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1293" already present on machine
22:24:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-ccbd696dd-w2mzz	Created	Created container: agent
22:24:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-ccbd696dd-w2mzz	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:24:58 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
22:24:58 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq	Unhealthy	Readiness probe failed: dial tcp 10.133.0.28:8080: connect: connection refused
22:25:01 (x4)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6c75bdff6f-cv6vq	Unhealthy	Readiness probe failed: Get "https://10.133.0.28:8643/healthz": dial tcp 10.133.0.28:8643: connect: connection refused
22:25:44 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:25:44 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:25:56 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:25:56	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	InferenceServiceReady	InferenceService [isvc-sklearn-batcher-custom] is Ready
22:25:59	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:25:59	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:26:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-ccbd696dd-w2mzz	Killing	Stopping container kserve-container
22:26:09	kserve-ci-e2e-test	replicaset-controller	message-dumper-predictor-c7d86bcbd	SuccessfulCreate	Created pod: message-dumper-predictor-c7d86bcbd-dqtxz
22:26:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-ccbd696dd-w2mzz	Killing	Stopping container agent
22:26:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-ccbd696dd-w2mzz	Killing	Stopping container kube-rbac-proxy
22:26:09	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-dqtxz	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "message-dumper-predictor-serving-cert" not found
22:26:09	kserve-ci-e2e-test	deployment-controller	message-dumper-predictor	ScalingReplicaSet	Scaled up replica set message-dumper-predictor-c7d86bcbd from 0 to 1
22:26:09	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	UpdateFailed	Failed to update status for InferenceService "message-dumper": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper": the object has been modified; please apply your changes to the latest version and try again
22:26:10	kserve-ci-e2e-test	multus	message-dumper-predictor-c7d86bcbd-dqtxz	AddedInterface	Add eth0 [10.133.0.30/23] from ovn-kubernetes
22:26:10	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-dqtxz	Pulling	Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display"
22:26:11	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-dqtxz	Started	Started container kserve-container
22:26:11	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-dqtxz	Started	Started container kube-rbac-proxy
22:26:11	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-dqtxz	Created	Created container: kube-rbac-proxy
22:26:11	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-dqtxz	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:26:11	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-dqtxz	Created	Created container: kserve-container
22:26:11	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-dqtxz	Pulled	Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.149s (1.149s including waiting). Image size: 14813193 bytes.
22:26:19	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	InferenceServiceReady	InferenceService [message-dumper] is Ready
22:26:19 (x8)	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:26:26 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-ccbd696dd-w2mzz	Unhealthy	Readiness probe failed: dial tcp 10.133.0.29:5000: connect: connection refused
22:26:26 (x4)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-ccbd696dd-w2mzz	Unhealthy	Readiness probe failed: Get "https://10.133.0.29:8643/healthz": dial tcp 10.133.0.29:8643: connect: connection refused
22:26:26 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-ccbd696dd-w2mzz	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
22:26:29	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	UpdateFailed	Failed to update status for InferenceService "isvc-logger": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger": the object has been modified; please apply your changes to the latest version and try again
22:26:29	kserve-ci-e2e-test	replicaset-controller	isvc-logger-predictor-64d54fcc88	SuccessfulCreate	Created pod: isvc-logger-predictor-64d54fcc88-bsbws
22:26:29	kserve-ci-e2e-test	multus	isvc-logger-predictor-64d54fcc88-bsbws	AddedInterface	Add eth0 [10.133.0.31/23] from ovn-kubernetes
22:26:29	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-64d54fcc88-bsbws	Created	Created container: storage-initializer
22:26:29	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-64d54fcc88-bsbws	Started	Started container storage-initializer
22:26:29	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-64d54fcc88-bsbws	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:26:29	kserve-ci-e2e-test	deployment-controller	isvc-logger-predictor	ScalingReplicaSet	Scaled up replica set isvc-logger-predictor-64d54fcc88 from 0 to 1
22:26:33	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-64d54fcc88-bsbws	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1293" already present on machine
22:26:33	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-64d54fcc88-bsbws	Created	Created container: kserve-container
22:26:34	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-64d54fcc88-bsbws	Created	Created container: kube-rbac-proxy
22:26:34	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-64d54fcc88-bsbws	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:cbf2f0964114aa89f1941320ac63f4a95a47d78bad8157d48396be0e3b5c023b" already present on machine
22:26:34	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-64d54fcc88-bsbws	Created	Created container: agent
22:26:34	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-64d54fcc88-bsbws	Started	Started container kube-rbac-proxy
22:26:34	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-64d54fcc88-bsbws	Started	Started container kserve-container
22:26:34	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-64d54fcc88-bsbws	Started	Started container agent
22:26:34	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-64d54fcc88-bsbws	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:26:39 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:26:39 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:26:54	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:26:54	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:27:14 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:27:14 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:27:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	InferenceServiceReady	InferenceService [isvc-logger] is Ready
22:27:42 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:27:44 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:27:44 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:27:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm": the object has been modified; please apply your changes to the latest version and try again
22:27:54	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-z5nft	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-lightgbm-predictor-serving-cert" not found
22:27:54	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-64d54fcc88-bsbws	Killing	Stopping container kube-rbac-proxy
22:27:54	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-64d54fcc88-bsbws	Killing	Stopping container kserve-container
22:27:54	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-64d54fcc88-bsbws	Killing	Stopping container agent
22:27:54	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-predictor-bdf964bd from 0 to 1
22:27:54	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-predictor-bdf964bd	SuccessfulCreate	Created pod: isvc-lightgbm-predictor-bdf964bd-z5nft
22:27:54	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-dqtxz	Killing	Stopping container kserve-container
22:27:54	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-dqtxz	Killing	Stopping container kube-rbac-proxy
22:27:55	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-z5nft	Started	Started container storage-initializer
22:27:55	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-z5nft	Created	Created container: storage-initializer
22:27:55	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-z5nft	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:27:55	kserve-ci-e2e-test	multus	isvc-lightgbm-predictor-bdf964bd-z5nft	AddedInterface	Add eth0 [10.133.0.32/23] from ovn-kubernetes
22:28:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-z5nft	Pulling	Pulling image "kserve/lgbserver:latest"
22:28:06	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-z5nft	Pulled	Successfully pulled image "kserve/lgbserver:latest" in 6.718s (6.718s including waiting). Image size: 606108943 bytes.
22:28:07	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-z5nft	Created	Created container: kube-rbac-proxy
22:28:07	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-z5nft	Started	Started container kube-rbac-proxy
22:28:07	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-z5nft	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:28:07	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-z5nft	Started	Started container kserve-container
22:28:07	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-z5nft	Created	Created container: kserve-container
22:28:11 (x10)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-64d54fcc88-bsbws	Unhealthy	Readiness probe failed: dial tcp 10.133.0.31:8080: connect: connection refused
22:28:11 (x10)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-64d54fcc88-bsbws	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
22:28:16 (x5)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-64d54fcc88-bsbws	Unhealthy	Readiness probe failed: Get "https://10.133.0.31:8643/healthz": dial tcp 10.133.0.31:8643: connect: connection refused
22:28:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:28:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:29:24 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:29:24 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:29:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	InferenceServiceReady	InferenceService [isvc-lightgbm] is Ready
22:29:28 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:29:34	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-z5nft	Killing	Stopping container kserve-container
22:29:34	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-runtime-predictor-749c4f6d58	SuccessfulCreate	Created pod: isvc-lightgbm-runtime-predictor-749c4f6d58-28jjm
22:29:34	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-z5nft	Killing	Stopping container kube-rbac-proxy
22:29:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-runtime": the object has been modified; please apply your changes to the latest version and try again
22:29:34 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-runtime": the object has been modified; please apply your changes to the latest version and try again
22:29:34	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-runtime-predictor-749c4f6d58 from 0 to 1
22:29:35	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-28jjm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:29:35	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-28jjm	Started	Started container storage-initializer
22:29:35	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-28jjm	Created	Created container: storage-initializer
22:29:35	kserve-ci-e2e-test	multus	isvc-lightgbm-runtime-predictor-749c4f6d58-28jjm	AddedInterface	Add eth0 [10.133.0.33/23] from ovn-kubernetes
22:29:38 (x11)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-z5nft	Unhealthy	Readiness probe failed: dial tcp 10.133.0.32:8080: connect: connection refused
22:29:39	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-28jjm	Pulled	Container image "kserve/lgbserver:latest" already present on machine
22:29:39	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-28jjm	Created	Created container: kserve-container
22:29:39	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-28jjm	Started	Started container kserve-container
22:29:39	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-28jjm	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:29:39	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-28jjm	Created	Created container: kube-rbac-proxy
22:29:39	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-28jjm	Started	Started container kube-rbac-proxy
22:29:40	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-z5nft	Unhealthy	Readiness probe failed: Get "https://10.133.0.32:8643/healthz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
22:30:04 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:30:04 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:30:56 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:30:56	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-runtime] is Ready
22:31:04 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:31:04 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:31:05 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
22:31:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-gmjt6	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
22:31:05	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-runtime-predictor-8765c9667 from 0 to 1
22:31:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-28jjm	Killing	Stopping container kserve-container
22:31:05	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-runtime-predictor-8765c9667	SuccessfulCreate	Created pod: isvc-lightgbm-v2-runtime-predictor-8765c9667-gmjt6
22:31:05	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
22:31:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-28jjm	Killing	Stopping container kube-rbac-proxy
22:31:06	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-gmjt6	Created	Created container: storage-initializer
22:31:06	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-runtime-predictor-8765c9667-gmjt6	AddedInterface	Add eth0 [10.133.0.34/23] from ovn-kubernetes
22:31:06	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-gmjt6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:31:06	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-gmjt6	Started	Started container storage-initializer
22:31:06 (x10)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-28jjm	Unhealthy	Readiness probe failed: dial tcp 10.133.0.33:8080: connect: connection refused
22:31:06	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-28jjm	Unhealthy	Readiness probe failed: Get "https://10.133.0.33:8643/healthz": dial tcp 10.133.0.33:8643: connect: connection refused
22:31:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-gmjt6	Pulling	Pulling image "docker.io/seldonio/mlserver:1.7.1"
22:33:24	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-gmjt6	Started	Started container kube-rbac-proxy
22:33:24	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-gmjt6	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:33:24	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-gmjt6	Created	Created container: kube-rbac-proxy
22:33:24	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-gmjt6	Started	Started container kserve-container
22:33:24	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-gmjt6	Created	Created container: kserve-container
22:33:24	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-gmjt6	Pulled	Successfully pulled image "docker.io/seldonio/mlserver:1.7.1" in 2m13.682s (2m13.682s including waiting). Image size: 10890461297 bytes.
22:33:35 (x10)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:33:35 (x10)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:34:01 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:34:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-runtime] is Ready
22:34:05	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-kserve-predictor-559bf6989	SuccessfulCreate	Created pod: isvc-lightgbm-v2-kserve-predictor-559bf6989-vdclq
22:34:05	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-kserve-predictor-559bf6989 from 0 to 1
22:34:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-gmjt6	Killing	Stopping container kube-rbac-proxy
22:34:05 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
22:34:05	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
22:34:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-vdclq	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
22:34:05 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:34:05 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:34:05	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-gmjt6	Killing	Stopping container kserve-container
22:34:06	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-vdclq	Created	Created container: storage-initializer
22:34:06	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-vdclq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:34:06	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-kserve-predictor-559bf6989-vdclq	AddedInterface	Add eth0 [10.133.0.35/23] from ovn-kubernetes
22:34:06	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-vdclq	Started	Started container storage-initializer
22:34:07	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-gmjt6	Unhealthy	Readiness probe failed: Get "https://10.133.0.34:8643/healthz": context deadline exceeded
22:34:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-vdclq	Started	Started container kserve-container
22:34:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-vdclq	Created	Created container: kserve-container
22:34:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-vdclq	Pulled	Container image "kserve/lgbserver:latest" already present on machine
22:34:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-vdclq	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:34:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-vdclq	Created	Created container: kube-rbac-proxy
22:34:10	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-vdclq	Started	Started container kube-rbac-proxy
22:34:13 (x2)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-vdclq	Unhealthy	Readiness probe failed: dial tcp 10.133.0.35:8080: connect: connection refused
22:34:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-kserve] is Ready
22:34:18 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:34:20	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:34:20	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:34:25	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-7l7dn	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
22:34:25	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-mlflow-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-mlflow-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
22:34:25	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-vdclq	Killing	Stopping container kube-rbac-proxy
22:34:25	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-vdclq	Killing	Stopping container kserve-container
22:34:25	kserve-ci-e2e-test	replicaset-controller	isvc-mlflow-v2-runtime-predictor-5fdb47d546	SuccessfulCreate	Created pod: isvc-mlflow-v2-runtime-predictor-5fdb47d546-7l7dn
22:34:25	kserve-ci-e2e-test	deployment-controller	isvc-mlflow-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-mlflow-v2-runtime-predictor-5fdb47d546 from 0 to 1
22:34:26	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-7l7dn	Created	Created container: storage-initializer
22:34:26	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-7l7dn	Started	Started container storage-initializer
22:34:26	kserve-ci-e2e-test	multus	isvc-mlflow-v2-runtime-predictor-5fdb47d546-7l7dn	AddedInterface	Add eth0 [10.133.0.36/23] from ovn-kubernetes
22:34:26	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-7l7dn	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:34:30	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-7l7dn	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:34:30	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-7l7dn	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
22:34:30	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-7l7dn	Created	Created container: kserve-container
22:34:30	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-7l7dn	Started	Started container kserve-container
22:34:30	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-7l7dn	Started	Started container kube-rbac-proxy
22:34:30	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-7l7dn	Created	Created container: kube-rbac-proxy
22:34:40	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:34:40	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:35:08 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:35:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	InferenceServiceReady	InferenceService [isvc-mlflow-v2-runtime] is Ready
22:35:10 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:35:10 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:35:15	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-mcp-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-mcp-predictor-5fdf4889b4 from 0 to 1
22:35:15	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-mcp-predictor-5fdf4889b4	SuccessfulCreate	Created pod: isvc-sklearn-mcp-predictor-5fdf4889b4-t2fgn
22:35:15	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-7l7dn	Killing	Stopping container kserve-container
22:35:15	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-7l7dn	Killing	Stopping container kube-rbac-proxy
22:35:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5fdf4889b4-t2fgn	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
22:35:15 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-mcp": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-mcp": the object has been modified; please apply your changes to the latest version and try again
22:35:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-mcp": the object has been modified; please apply your changes to the latest version and try again
22:35:16	kserve-ci-e2e-test	multus	isvc-sklearn-mcp-predictor-5fdf4889b4-t2fgn	AddedInterface	Add eth0 [10.133.0.37/23] from ovn-kubernetes
22:35:16	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5fdf4889b4-t2fgn	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:35:16	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5fdf4889b4-t2fgn	Created	Created container: storage-initializer
22:35:16	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5fdf4889b4-t2fgn	Started	Started container storage-initializer
22:35:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5fdf4889b4-t2fgn	Started	Started container kserve-container
22:35:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5fdf4889b4-t2fgn	Pulling	Pulling image "quay.io/opendatahub/kserve-agent:latest"
22:35:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5fdf4889b4-t2fgn	Created	Created container: kserve-container
22:35:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5fdf4889b4-t2fgn	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1293" already present on machine
22:35:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5fdf4889b4-t2fgn	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent:latest" in 2.244s (2.244s including waiting). Image size: 237663782 bytes.
22:35:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5fdf4889b4-t2fgn	Created	Created container: kserve-agent
22:35:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5fdf4889b4-t2fgn	Started	Started container kserve-agent
22:35:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5fdf4889b4-t2fgn	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:35:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5fdf4889b4-t2fgn	Created	Created container: kube-rbac-proxy
22:35:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5fdf4889b4-t2fgn	Started	Started container kube-rbac-proxy
22:36:29	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	InferenceServiceReady	InferenceService [isvc-sklearn-mcp] is Ready
22:36:29 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:36:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5fdf4889b4-t2fgn	Killing	Stopping container kube-rbac-proxy
22:36:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5fdf4889b4-t2fgn	Killing	Stopping container kserve-container
22:36:35	kserve-ci-e2e-test	deployment-controller	isvc-paddle-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-predictor-6b8b7cfb4b from 0 to 1
22:36:35	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-predictor-6b8b7cfb4b	SuccessfulCreate	Created pod: isvc-paddle-predictor-6b8b7cfb4b-6hx29
22:36:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5fdf4889b4-t2fgn	Killing	Stopping container kserve-agent
22:36:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	UpdateFailed	Failed to update status for InferenceService "isvc-paddle": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle": the object has been modified; please apply your changes to the latest version and try again
22:36:36	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-6hx29	Started	Started container storage-initializer
22:36:36	kserve-ci-e2e-test	multus	isvc-paddle-predictor-6b8b7cfb4b-6hx29	AddedInterface	Add eth0 [10.133.0.38/23] from ovn-kubernetes
22:36:36	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-6hx29	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:36:36	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-6hx29	Created	Created container: storage-initializer
22:36:43	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-6hx29	Pulling	Pulling image "kserve/paddleserver:latest"
22:36:54	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-6hx29	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:36:54	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-6hx29	Pulled	Successfully pulled image "kserve/paddleserver:latest" in 11.409s (11.409s including waiting). Image size: 1162639611 bytes.
22:36:54	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-6hx29	Started	Started container kube-rbac-proxy
22:36:54	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-6hx29	Created	Created container: kube-rbac-proxy
22:36:54	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-6hx29	Created	Created container: kserve-container
22:36:54	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-6hx29	Started	Started container kserve-container
22:36:59 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5fdf4889b4-t2fgn	Unhealthy	Readiness probe failed: Get "http://10.133.0.37:8080/v1/models/isvc-sklearn-mcp": dial tcp 10.133.0.37:8080: connect: connection refused
22:37:04 (x6)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5fdf4889b4-t2fgn	Unhealthy	Readiness probe failed: Get "https://10.133.0.37:8643/healthz": dial tcp 10.133.0.37:8643: connect: connection refused
22:37:05 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:37:05 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:37:32 (x6)	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-6hx29	Unhealthy	Readiness probe failed: dial tcp 10.133.0.38:8080: connect: connection refused
22:37:35 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:37:35 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:37:42 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:37:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	InferenceServiceReady	InferenceService [isvc-paddle] is Ready
22:37:47	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-6hx29	Killing	Stopping container kube-rbac-proxy
22:37:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-runtime": the object has been modified; please apply your changes to the latest version and try again
22:37:47	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-6hx29	Killing	Stopping container kserve-container
22:37:47	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-m76wv	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-paddle-runtime-predictor-serving-cert" not found
22:37:47	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-runtime-predictor-7f4d4f9dc8	SuccessfulCreate	Created pod: isvc-paddle-runtime-predictor-7f4d4f9dc8-m76wv
22:37:47	kserve-ci-e2e-test	deployment-controller	isvc-paddle-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-runtime-predictor-7f4d4f9dc8 from 0 to 1
22:37:47 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-runtime": the object has been modified; please apply your changes to the latest version and try again
22:37:48	kserve-ci-e2e-test	multus	isvc-paddle-runtime-predictor-7f4d4f9dc8-m76wv	AddedInterface	Add eth0 [10.133.0.39/23] from ovn-kubernetes
22:37:48	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-m76wv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:37:48	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-m76wv	Created	Created container: storage-initializer
22:37:48	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-m76wv	Started	Started container storage-initializer
22:37:53	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-m76wv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:37:53	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-m76wv	Pulled	Container image "kserve/paddleserver:latest" already present on machine
22:37:53	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-m76wv	Created	Created container: kserve-container
22:37:53	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-m76wv	Started	Started container kserve-container
22:37:53	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-m76wv	Created	Created container: kube-rbac-proxy
22:37:53	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-m76wv	Started	Started container kube-rbac-proxy
22:38:17 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:38:17 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:38:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	InferenceServiceReady	InferenceService [isvc-paddle-runtime] is Ready
22:38:40 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:38:47 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:38:47 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:38:48	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-v2-kserve-predictor-7dbd59854	SuccessfulCreate	Created pod: isvc-paddle-v2-kserve-predictor-7dbd59854-t4fqm
22:38:48	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
22:38:48	kserve-ci-e2e-test	deployment-controller	isvc-paddle-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-v2-kserve-predictor-7dbd59854 from 0 to 1
22:38:48 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
22:38:48	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-m76wv	Killing	Stopping container kserve-container
22:38:48	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-m76wv	Killing	Stopping container kube-rbac-proxy
22:38:49	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-t4fqm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:38:49	kserve-ci-e2e-test	multus	isvc-paddle-v2-kserve-predictor-7dbd59854-t4fqm	AddedInterface	Add eth0 [10.133.0.40/23] from ovn-kubernetes
22:38:49	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-t4fqm	Created	Created container: storage-initializer
22:38:49	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-t4fqm	Started	Started container storage-initializer
22:38:50	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-m76wv	Unhealthy	Readiness probe failed: Get "https://10.133.0.39:8643/healthz": dial tcp 10.133.0.39:8643: connect: connection refused
22:38:50 (x7)	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-m76wv	Unhealthy	Readiness probe failed: dial tcp 10.133.0.39:8080: connect: connection refused
22:38:54	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-t4fqm	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:38:54	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-t4fqm	Started	Started container kserve-container
22:38:54	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-t4fqm	Created	Created container: kserve-container
22:38:54	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-t4fqm	Created	Created container: kube-rbac-proxy
22:38:54	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-t4fqm	Started	Started container kube-rbac-proxy
22:38:54	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-t4fqm	Pulled	Container image "kserve/paddleserver:latest" already present on machine
22:39:33 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:39:33 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:39:42 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:39:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	InferenceServiceReady	InferenceService [isvc-paddle-v2-kserve] is Ready
22:39:48	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:39:48	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:39:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	UpdateFailed	Failed to update status for InferenceService "isvc-pmml": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml": the object has been modified; please apply your changes to the latest version and try again
22:39:50	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-t4fqm	Killing	Stopping container kube-rbac-proxy
22:39:50	kserve-ci-e2e-test	deployment-controller	isvc-pmml-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-predictor-8bb578669 from 0 to 1
22:39:50	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-predictor-8bb578669	SuccessfulCreate	Created pod: isvc-pmml-predictor-8bb578669-p57jg
22:39:50	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-t4fqm	Killing	Stopping container kserve-container
22:39:50	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-p57jg	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-pmml-predictor-serving-cert" not found
22:39:51	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-p57jg	Created	Created container: storage-initializer
22:39:51	kserve-ci-e2e-test	multus	isvc-pmml-predictor-8bb578669-p57jg	AddedInterface	Add eth0 [10.133.0.41/23] from ovn-kubernetes
22:39:51	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-p57jg	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:39:51	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-p57jg	Started	Started container storage-initializer
22:39:52 (x7)	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-t4fqm	Unhealthy	Readiness probe failed: dial tcp 10.133.0.40:8080: connect: connection refused
22:39:52	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-t4fqm	Unhealthy	Readiness probe failed: Get "https://10.133.0.40:8643/healthz": dial tcp 10.133.0.40:8643: connect: connection refused
22:39:55	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-p57jg	Pulling	Pulling image "kserve/pmmlserver:latest"
22:40:02	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-p57jg	Created	Created container: kserve-container
22:40:02	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-p57jg	Pulled	Successfully pulled image "kserve/pmmlserver:latest" in 6.973s (6.973s including waiting). Image size: 800715639 bytes.
22:40:02	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-p57jg	Started	Started container kserve-container
22:40:02	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-p57jg	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:40:02	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-p57jg	Created	Created container: kube-rbac-proxy
22:40:02	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-p57jg	Started	Started container kube-rbac-proxy
22:40:05	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:40:05	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:41:19 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-p57jg	Unhealthy	Readiness probe failed: dial tcp 10.133.0.41:8080: connect: connection refused
22:41:20 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:41:20 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:41:29 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:41:29	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	InferenceServiceReady	InferenceService [isvc-pmml] is Ready
22:41:32	kserve-ci-e2e-test	deployment-controller	isvc-pmml-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-runtime-predictor-67bc544947 from 0 to 1
22:41:32	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-runtime-predictor-67bc544947	SuccessfulCreate	Created pod: isvc-pmml-runtime-predictor-67bc544947-dkqsr
22:41:32	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-dkqsr	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-pmml-runtime-predictor-serving-cert" not found
22:41:32	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-p57jg	Killing	Stopping container kube-rbac-proxy
22:41:32	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-runtime": the object has been modified; please apply your changes to the latest version and try again
22:41:32	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-p57jg	Killing	Stopping container kserve-container
22:41:33	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-dkqsr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:41:33	kserve-ci-e2e-test	multus	isvc-pmml-runtime-predictor-67bc544947-dkqsr	AddedInterface	Add eth0 [10.133.0.42/23] from ovn-kubernetes
22:41:33	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-dkqsr	Started	Started container storage-initializer
22:41:33	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-dkqsr	Created	Created container: storage-initializer
22:41:34	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-p57jg	Unhealthy	Readiness probe failed: Get "https://10.133.0.41:8643/healthz": dial tcp 10.133.0.41:8643: connect: connection refused
22:41:37	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-dkqsr	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
22:41:37	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-dkqsr	Created	Created container: kserve-container
22:41:37	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-dkqsr	Started	Started container kserve-container
22:41:37	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-dkqsr	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:41:37	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-dkqsr	Created	Created container: kube-rbac-proxy
22:41:37	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-dkqsr	Started	Started container kube-rbac-proxy
22:42:17 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:42:17 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:42:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	InferenceServiceReady	InferenceService [isvc-pmml-runtime] is Ready
22:42:55 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:43:02 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:43:02 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:43:03	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-v2-kserve-predictor-6578f8ffc7	SuccessfulCreate	Created pod: isvc-pmml-v2-kserve-predictor-6578f8ffc7-vzp5s
22:43:03	kserve-ci-e2e-test	multus	isvc-pmml-v2-kserve-predictor-6578f8ffc7-vzp5s	AddedInterface	Add eth0 [10.133.0.43/23] from ovn-kubernetes
22:43:03	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-vzp5s	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:43:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
22:43:03	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-vzp5s	Started	Started container storage-initializer
22:43:03	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-dkqsr	Killing	Stopping container kube-rbac-proxy
22:43:03	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-dkqsr	Killing	Stopping container kserve-container
22:43:03	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-vzp5s	Created	Created container: storage-initializer
22:43:03	kserve-ci-e2e-test	deployment-controller	isvc-pmml-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-v2-kserve-predictor-6578f8ffc7 from 0 to 1
22:43:05 (x11)	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-dkqsr	Unhealthy	Readiness probe failed: dial tcp 10.133.0.42:8080: connect: connection refused
22:43:05	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-dkqsr	Unhealthy	Readiness probe failed: Get "https://10.133.0.42:8643/healthz": dial tcp 10.133.0.42:8643: connect: connection refused
22:43:08	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-vzp5s	Started	Started container kube-rbac-proxy
22:43:08	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-vzp5s	Created	Created container: kube-rbac-proxy
22:43:08	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-vzp5s	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:43:08	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-vzp5s	Started	Started container kserve-container
22:43:08	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-vzp5s	Created	Created container: kserve-container
22:43:08	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-vzp5s	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
22:43:18	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:43:18	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:44:26 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:44:26	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	InferenceServiceReady	InferenceService [isvc-pmml-v2-kserve] is Ready
22:44:33 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:44:33 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:44:34	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-vzp5s	Killing	Stopping container kube-rbac-proxy
22:44:34	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-vzp5s	Killing	Stopping container kserve-container
22:44:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-0d7654	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-0d7654": the object has been modified; please apply your changes to the latest version and try again
22:44:34	kserve-ci-e2e-test	deployment-controller	isvc-primary-0d7654-predictor	ScalingReplicaSet	Scaled up replica set isvc-primary-0d7654-predictor-ff757f45b from 0 to 1
22:44:34	kserve-ci-e2e-test	kubelet	isvc-primary-0d7654-predictor-ff757f45b-xphvm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:44:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-0d7654	UpdateFailed	Failed to update status for InferenceService "isvc-primary-0d7654": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-0d7654": the object has been modified; please apply your changes to the latest version and try again
22:44:34	kserve-ci-e2e-test	multus	isvc-primary-0d7654-predictor-ff757f45b-xphvm	AddedInterface	Add eth0 [10.133.0.44/23] from ovn-kubernetes
22:44:34	kserve-ci-e2e-test	replicaset-controller	isvc-primary-0d7654-predictor-ff757f45b	SuccessfulCreate	Created pod: isvc-primary-0d7654-predictor-ff757f45b-xphvm
22:44:35	kserve-ci-e2e-test	kubelet	isvc-primary-0d7654-predictor-ff757f45b-xphvm	Created	Created container: storage-initializer
22:44:35	kserve-ci-e2e-test	kubelet	isvc-primary-0d7654-predictor-ff757f45b-xphvm	Started	Started container storage-initializer
22:44:36 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-vzp5s	Unhealthy	Readiness probe failed: dial tcp 10.133.0.43:8080: connect: connection refused
22:44:36	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-vzp5s	Unhealthy	Readiness probe failed: Get "https://10.133.0.43:8643/healthz": dial tcp 10.133.0.43:8643: connect: connection refused
22:44:39	kserve-ci-e2e-test	kubelet	isvc-primary-0d7654-predictor-ff757f45b-xphvm	Created	Created container: kserve-container
22:44:39	kserve-ci-e2e-test	kubelet	isvc-primary-0d7654-predictor-ff757f45b-xphvm	Started	Started container kserve-container
22:44:39	kserve-ci-e2e-test	kubelet	isvc-primary-0d7654-predictor-ff757f45b-xphvm	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:44:39	kserve-ci-e2e-test	kubelet	isvc-primary-0d7654-predictor-ff757f45b-xphvm	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1293" already present on machine
22:44:39	kserve-ci-e2e-test	kubelet	isvc-primary-0d7654-predictor-ff757f45b-xphvm	Created	Created container: kube-rbac-proxy
22:44:39	kserve-ci-e2e-test	kubelet	isvc-primary-0d7654-predictor-ff757f45b-xphvm	Started	Started container kube-rbac-proxy
22:44:49	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-0d7654-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:44:49	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-0d7654-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:45:37 (x8)	kserve-ci-e2e-test	kubelet	isvc-primary-0d7654-predictor-ff757f45b-xphvm	Unhealthy	Readiness probe failed: dial tcp 10.133.0.44:8080: connect: connection refused
22:45:47 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-0d7654	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:45:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-0d7654	InferenceServiceReady	InferenceService [isvc-primary-0d7654] is Ready
22:45:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-0d7654	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-0d7654": the object has been modified; please apply your changes to the latest version and try again
22:45:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-0d7654	UpdateFailed	Failed to update status for InferenceService "isvc-secondary-0d7654": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-0d7654": the object has been modified; please apply your changes to the latest version and try again
22:45:54	kserve-ci-e2e-test	deployment-controller	isvc-secondary-0d7654-predictor	ScalingReplicaSet	Scaled up replica set isvc-secondary-0d7654-predictor-6647fcc567 from 0 to 1
22:45:54	kserve-ci-e2e-test	replicaset-controller	isvc-secondary-0d7654-predictor-6647fcc567	SuccessfulCreate	Created pod: isvc-secondary-0d7654-predictor-6647fcc567-bpj6r
22:45:55	kserve-ci-e2e-test	multus	isvc-secondary-0d7654-predictor-6647fcc567-bpj6r	AddedInterface	Add eth0 [10.133.0.45/23] from ovn-kubernetes
22:45:59 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-0d7654-predictor-6647fcc567-bpj6r	Started	Started container storage-initializer
22:45:59 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-0d7654-predictor-6647fcc567-bpj6r	Created	Created container: storage-initializer
22:45:59 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-0d7654-predictor-6647fcc567-bpj6r	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:46:02	kserve-ci-e2e-test	kubelet	isvc-secondary-0d7654-predictor-6647fcc567-bpj6r	BackOff	Back-off restarting failed container storage-initializer in pod isvc-secondary-0d7654-predictor-6647fcc567-bpj6r_kserve-ci-e2e-test(4a502f95-1664-41fd-9531-71b9492f8b69)
22:46:03 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-0d7654	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:46:04 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-0d7654-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:46:04 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-0d7654-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:46:09	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-0d7654-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:46:09	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-0d7654-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:46:10	kserve-ci-e2e-test	replicaset-controller	isvc-init-fail-241fac-predictor-79dffbcd5f	SuccessfulCreate	Created pod: isvc-init-fail-241fac-predictor-79dffbcd5f-c5zcg
22:46:10	kserve-ci-e2e-test	kubelet	isvc-init-fail-241fac-predictor-79dffbcd5f-c5zcg	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-init-fail-241fac-predictor-serving-cert" not found
22:46:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-241fac	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-241fac": the object has been modified; please apply your changes to the latest version and try again
22:46:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-241fac	UpdateFailed	Failed to update status for InferenceService "isvc-init-fail-241fac": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-241fac": the object has been modified; please apply your changes to the latest version and try again
22:46:10	kserve-ci-e2e-test	deployment-controller	isvc-init-fail-241fac-predictor	ScalingReplicaSet	Scaled up replica set isvc-init-fail-241fac-predictor-79dffbcd5f from 0 to 1
22:46:10	kserve-ci-e2e-test	kubelet	isvc-primary-0d7654-predictor-ff757f45b-xphvm	Killing	Stopping container kserve-container
22:46:10	kserve-ci-e2e-test	kubelet	isvc-primary-0d7654-predictor-ff757f45b-xphvm	Killing	Stopping container kube-rbac-proxy
22:46:11	kserve-ci-e2e-test	multus	isvc-init-fail-241fac-predictor-79dffbcd5f-c5zcg	AddedInterface	Add eth0 [10.133.0.46/23] from ovn-kubernetes
22:46:12	kserve-ci-e2e-test	kubelet	isvc-primary-0d7654-predictor-ff757f45b-xphvm	Unhealthy	Readiness probe failed: Get "https://10.133.0.44:8643/healthz": dial tcp 10.133.0.44:8643: connect: connection refused
22:46:16 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-241fac-predictor-79dffbcd5f-c5zcg	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:46:17 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-241fac-predictor-79dffbcd5f-c5zcg	Created	Created container: storage-initializer
22:46:17 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-241fac	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:46:17 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-241fac-predictor-79dffbcd5f-c5zcg	Started	Started container storage-initializer
22:46:20	kserve-ci-e2e-test	kubelet	isvc-init-fail-241fac-predictor-79dffbcd5f-c5zcg	Killing	Stopping container storage-initializer
22:46:20	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-predictor-cd7c759c9	SuccessfulCreate	Created pod: isvc-predictive-sklearn-predictor-cd7c759c9-hqwrv
22:46:20	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-predictor-cd7c759c9 from 0 to 1
22:46:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn": the object has been modified; please apply your changes to the latest version and try again
22:46:21	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-predictor-cd7c759c9-hqwrv	AddedInterface	Add eth0 [10.133.0.47/23] from ovn-kubernetes
22:46:21	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-hqwrv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:46:21	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-hqwrv	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-predictive-sklearn-predictor-serving-cert" not found
22:46:22	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-hqwrv	Created	Created container: storage-initializer
22:46:22	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-hqwrv	Started	Started container storage-initializer
22:46:25	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-hqwrv	Pulling	Pulling image "kserve/predictiveserver:latest"
22:46:45	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-hqwrv	Pulled	Successfully pulled image "kserve/predictiveserver:latest" in 19.757s (19.757s including waiting). Image size: 2312633199 bytes.
22:46:45	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-hqwrv	Started	Started container kserve-container
22:46:45	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-hqwrv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:46:45	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-hqwrv	Created	Created container: kserve-container
22:46:46	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-hqwrv	Created	Created container: kube-rbac-proxy
22:46:46	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-hqwrv	Started	Started container kube-rbac-proxy
22:47:20 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:47:20 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:48:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	InferenceServiceReady	InferenceService [isvc-predictive-sklearn] is Ready
22:48:03 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:48:05 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:48:05 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:48:11	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-fhw8t	Created	Created container: storage-initializer
22:48:11	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-predictor-7ff98fd74d-fhw8t	AddedInterface	Add eth0 [10.133.0.48/23] from ovn-kubernetes
22:48:11	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-hqwrv	Killing	Stopping container kube-rbac-proxy
22:48:11	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-hqwrv	Killing	Stopping container kserve-container
22:48:11	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-fhw8t	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:48:11	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-fhw8t	Started	Started container storage-initializer
22:48:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost": the object has been modified; please apply your changes to the latest version and try again
22:48:11	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-predictor-7ff98fd74d from 0 to 1
22:48:11	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-predictor-7ff98fd74d	SuccessfulCreate	Created pod: isvc-predictive-xgboost-predictor-7ff98fd74d-fhw8t
22:48:13 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-hqwrv	Unhealthy	Readiness probe failed: dial tcp 10.133.0.47:8080: connect: connection refused
22:48:13	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-hqwrv	Unhealthy	Readiness probe failed: Get "https://10.133.0.47:8643/healthz": dial tcp 10.133.0.47:8643: connect: connection refused
22:48:15	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-fhw8t	Started	Started container kube-rbac-proxy
22:48:15	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-fhw8t	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
22:48:15	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-fhw8t	Created	Created container: kube-rbac-proxy
22:48:15	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-fhw8t	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:48:15	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-fhw8t	Started	Started container kserve-container
22:48:15	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-fhw8t	Created	Created container: kserve-container
22:48:56 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:48:56 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:49:27 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-fhw8t	Unhealthy	Readiness probe failed: dial tcp 10.133.0.48:8080: connect: connection refused
22:49:37 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:49:37	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	InferenceServiceReady	InferenceService [isvc-predictive-xgboost] is Ready
22:49:41	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-fhw8t	Killing	Stopping container kserve-container
22:49:41	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-fhw8t	Killing	Stopping container kube-rbac-proxy
22:49:41	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-hwz4l	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:49:41	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-hwz4l	Created	Created container: storage-initializer
22:49:41	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-hwz4l	Started	Started container storage-initializer
22:49:41	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-predictor-75cb94f9f from 0 to 1
22:49:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm": the object has been modified; please apply your changes to the latest version and try again
22:49:41	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-predictor-75cb94f9f-hwz4l	AddedInterface	Add eth0 [10.133.0.49/23] from ovn-kubernetes
22:49:41 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:49:41	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-predictor-75cb94f9f	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-predictor-75cb94f9f-hwz4l
22:49:41 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:49:42	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-fhw8t	Unhealthy	Readiness probe failed: Get "https://10.133.0.48:8643/healthz": dial tcp 10.133.0.48:8643: connect: connection refused
22:49:46	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-hwz4l	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
22:49:46	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-hwz4l	Started	Started container kube-rbac-proxy
22:49:46	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-hwz4l	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:49:46	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-hwz4l	Created	Created container: kserve-container
22:49:46	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-hwz4l	Created	Created container: kube-rbac-proxy
22:49:46	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-hwz4l	Started	Started container kserve-container
22:50:26 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:50:26 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:51:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm] is Ready
22:51:03 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:51:11	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-8ns7j	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
22:51:11 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:51:11 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:51:11	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-v2-predictor-b5d4f6b79	SuccessfulCreate	Created pod: isvc-predictive-sklearn-v2-predictor-b5d4f6b79-8ns7j
22:51:11	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-v2-predictor-b5d4f6b79 from 0 to 1
22:51:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
22:51:11	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-hwz4l	Killing	Stopping container kube-rbac-proxy
22:51:11	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-hwz4l	Killing	Stopping container kserve-container
22:51:12	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-8ns7j	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:51:12	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-8ns7j	Started	Started container storage-initializer
22:51:12	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-8ns7j	Created	Created container: storage-initializer
22:51:12	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-8ns7j	AddedInterface	Add eth0 [10.133.0.50/23] from ovn-kubernetes
22:51:13 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-hwz4l	Unhealthy	Readiness probe failed: dial tcp 10.133.0.49:8080: connect: connection refused
22:51:13	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-hwz4l	Unhealthy	Readiness probe failed: Get "https://10.133.0.49:8643/healthz": dial tcp 10.133.0.49:8643: connect: connection refused
22:51:16	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-8ns7j	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
22:51:17	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-8ns7j	Started	Started container kube-rbac-proxy
22:51:17	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-8ns7j	Created	Created container: kube-rbac-proxy
22:51:17	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-8ns7j	Created	Created container: kserve-container
22:51:17	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-8ns7j	Started	Started container kserve-container
22:51:17	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-8ns7j	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:51:26	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:51:26	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:52:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	InferenceServiceReady	InferenceService [isvc-predictive-sklearn-v2] is Ready
22:52:34 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:52:41	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-v2-predictor-6577c65fd8	SuccessfulCreate	Created pod: isvc-predictive-xgboost-v2-predictor-6577c65fd8-5sqbb
22:52:41 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:52:41 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
22:52:41	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-8ns7j	Killing	Stopping container kserve-container
22:52:41	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-8ns7j	Killing	Stopping container kube-rbac-proxy
22:52:41	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-v2-predictor-6577c65fd8 from 0 to 1
22:52:41 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:52:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
22:52:42	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-v2-predictor-6577c65fd8-5sqbb	AddedInterface	Add eth0 [10.133.0.51/23] from ovn-kubernetes
22:52:42	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-5sqbb	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:52:42	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-5sqbb	Created	Created container: storage-initializer
22:52:42	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-5sqbb	Started	Started container storage-initializer
22:52:43	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-8ns7j	Unhealthy	Readiness probe failed: Get "https://10.133.0.50:8643/healthz": dial tcp 10.133.0.50:8643: connect: connection refused
22:52:44 (x6)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-8ns7j	Unhealthy	Readiness probe failed: Get "http://10.133.0.50:8080/v2/models/isvc-predictive-sklearn-v2/ready": dial tcp 10.133.0.50:8080: connect: connection refused
22:52:46	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-5sqbb	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
22:52:46	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-5sqbb	Started	Started container kserve-container
22:52:46	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-5sqbb	Created	Created container: kserve-container
22:52:46	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-5sqbb	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:52:46	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-5sqbb	Created	Created container: kube-rbac-proxy
22:52:46	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-5sqbb	Started	Started container kube-rbac-proxy
22:53:26 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:53:26 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:54:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	InferenceServiceReady	InferenceService [isvc-predictive-xgboost-v2] is Ready
22:54:04 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:54:11	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-5sqbb	Killing	Stopping container kube-rbac-proxy
22:54:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm-v2": the object has been modified; please apply your changes to the latest version and try again
22:54:11	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-v2-predictor-865b4598f7 from 0 to 1
22:54:11	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-v2-predictor-865b4598f7	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-v2-predictor-865b4598f7-t58bp
22:54:11	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-5sqbb	Killing	Stopping container kserve-container
22:54:11 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:54:11 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:54:12	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-v2-predictor-865b4598f7-t58bp	AddedInterface	Add eth0 [10.133.0.52/23] from ovn-kubernetes
22:54:12	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-t58bp	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
22:54:13	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-t58bp	Created	Created container: storage-initializer
22:54:13	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-t58bp	Started	Started container storage-initializer
22:54:13	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-t58bp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:54:14 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-5sqbb	Unhealthy	Readiness probe failed: Get "http://10.133.0.51:8080/v2/models/isvc-predictive-xgboost-v2/ready": dial tcp 10.133.0.51:8080: connect: connection refused
22:54:14	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-5sqbb	Unhealthy	Readiness probe failed: Get "https://10.133.0.51:8643/healthz": dial tcp 10.133.0.51:8643: connect: connection refused
22:54:17	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-t58bp	Created	Created container: kserve-container
22:54:17	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-t58bp	Started	Started container kube-rbac-proxy
22:54:17	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-t58bp	Created	Created container: kube-rbac-proxy
22:54:17	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-t58bp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:54:17	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-t58bp	Started	Started container kserve-container
22:54:17	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-t58bp	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
22:54:56 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:54:56 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:55:34 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:55:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm-v2] is Ready
22:55:41 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:55:41 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:55:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
22:55:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-scheduler": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
22:55:42	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-t58bp	Killing	Stopping container kube-rbac-proxy
22:55:42	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-t58bp	Killing	Stopping container kserve-container
22:55:42 (x5)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:55:42	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-scheduler-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-scheduler-predictor-c477977d5 from 0 to 1
22:55:42	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-scheduler-predictor-c477977d5	SuccessfulCreate	Created pod: isvc-sklearn-scheduler-predictor-c477977d5-zv5hv
22:55:44 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-t58bp	Unhealthy	Readiness probe failed: Get "http://10.133.0.52:8080/v2/models/isvc-predictive-lightgbm-v2/ready": dial tcp 10.133.0.52:8080: connect: connection refused
22:55:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d8dbfbbb9-j5mxm	Started	Started container storage-initializer
22:55:44	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-predictor-d8dbfbbb9 from 0 to 1
22:55:44	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-predictor-d8dbfbbb9	SuccessfulCreate	Created pod: isvc-sklearn-predictor-d8dbfbbb9-j5mxm
22:55:44	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-t58bp	Unhealthy	Readiness probe failed: Get "https://10.133.0.52:8643/healthz": dial tcp 10.133.0.52:8643: connect: connection refused
22:55:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d8dbfbbb9-j5mxm	Created	Created container: storage-initializer
22:55:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn": the object has been modified; please apply your changes to the latest version and try again
22:55:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d8dbfbbb9-j5mxm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:55:44	kserve-ci-e2e-test	multus	isvc-sklearn-predictor-d8dbfbbb9-j5mxm	AddedInterface	Add eth0 [10.133.0.53/23] from ovn-kubernetes
22:55:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d8dbfbbb9-j5mxm	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1293" already present on machine
22:55:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d8dbfbbb9-j5mxm	Created	Created container: kserve-container
22:55:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d8dbfbbb9-j5mxm	Started	Started container kserve-container
22:55:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d8dbfbbb9-j5mxm	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:55:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d8dbfbbb9-j5mxm	Created	Created container: kube-rbac-proxy
22:55:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d8dbfbbb9-j5mxm	Started	Started container kube-rbac-proxy
22:56:29 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:56:29 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:56:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	InferenceServiceReady	InferenceService [isvc-sklearn] is Ready
22:56:55 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:56:59 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:56:59 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:57:04	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "sklearn-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
22:57:04 (x2)	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "sklearn-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "sklearn-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
22:57:04	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-f4fdh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:57:04	kserve-ci-e2e-test	multus	sklearn-v2-mlserver-predictor-65d8664766-f4fdh	AddedInterface	Add eth0 [10.133.0.54/23] from ovn-kubernetes
22:57:04	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-f4fdh	Started	Started container storage-initializer
22:57:04	kserve-ci-e2e-test	replicaset-controller	sklearn-v2-mlserver-predictor-65d8664766	SuccessfulCreate	Created pod: sklearn-v2-mlserver-predictor-65d8664766-f4fdh
22:57:04	kserve-ci-e2e-test	deployment-controller	sklearn-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set sklearn-v2-mlserver-predictor-65d8664766 from 0 to 1
22:57:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d8dbfbbb9-j5mxm	Killing	Stopping container kube-rbac-proxy
22:57:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d8dbfbbb9-j5mxm	Killing	Stopping container kserve-container
22:57:04	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-f4fdh	Created	Created container: storage-initializer
22:57:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d8dbfbbb9-j5mxm	Unhealthy	Readiness probe failed: Get "https://10.133.0.53:8643/healthz": dial tcp 10.133.0.53:8643: connect: connection refused
22:57:05 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-d8dbfbbb9-j5mxm	Unhealthy	Readiness probe failed: dial tcp 10.133.0.53:8080: connect: connection refused
22:57:09	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-f4fdh	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
22:57:09	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-f4fdh	Created	Created container: kube-rbac-proxy
22:57:09	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-f4fdh	Started	Started container kube-rbac-proxy
22:57:09	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-f4fdh	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:57:09	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-f4fdh	Started	Started container kserve-container
22:57:09	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-f4fdh	Created	Created container: kserve-container
22:57:19	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:57:19	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:57:46	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-f4fdh	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
22:57:56 (x12)	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:57:56	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	InferenceServiceReady	InferenceService [sklearn-v2-mlserver] is Ready
22:58:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-runtime": the object has been modified; please apply your changes to the latest version and try again
22:58:04 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:58:04	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-f4fdh	Killing	Stopping container kserve-container
22:58:04	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-f4fdh	Killing	Stopping container kube-rbac-proxy
22:58:04 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:58:04	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-runtime-predictor-65cd49579f from 0 to 1
22:58:04	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-runtime-predictor-65cd49579f	SuccessfulCreate	Created pod: isvc-sklearn-runtime-predictor-65cd49579f-dxsln
22:58:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-65cd49579f-dxsln	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
22:58:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-65cd49579f-dxsln	Started	Started container storage-initializer
22:58:05	kserve-ci-e2e-test	multus	isvc-sklearn-runtime-predictor-65cd49579f-dxsln	AddedInterface	Add eth0 [10.133.0.55/23] from ovn-kubernetes
22:58:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-65cd49579f-dxsln	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:58:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-65cd49579f-dxsln	Created	Created container: storage-initializer
22:58:07	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-f4fdh	Unhealthy	Readiness probe failed: Get "http://10.133.0.54:8080/v2/models/sklearn-v2-mlserver/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
22:58:11 (x2)	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-f4fdh	Unhealthy	Readiness probe failed: Get "https://10.133.0.54:8643/healthz": dial tcp 10.133.0.54:8643: connect: connection refused
22:58:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-65cd49579f-dxsln	Started	Started container kube-rbac-proxy
22:58:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-65cd49579f-dxsln	Created	Created container: kube-rbac-proxy
22:58:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-65cd49579f-dxsln	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:58:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-65cd49579f-dxsln	Started	Started container kserve-container
22:58:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-65cd49579f-dxsln	Created	Created container: kserve-container
22:58:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-65cd49579f-dxsln	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1293" already present on machine
22:58:18 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-65cd49579f-dxsln	Unhealthy	Readiness probe failed: dial tcp 10.133.0.55:8080: connect: connection refused
22:58:28 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:58:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-runtime] is Ready
22:58:34 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:58:34 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:58:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
22:58:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-65cd49579f-dxsln	Killing	Stopping container kserve-container
22:58:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-65cd49579f-dxsln	Killing	Stopping container kube-rbac-proxy
22:58:41	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-runtime-predictor-6d84c876f4	SuccessfulCreate	Created pod: isvc-sklearn-v2-runtime-predictor-6d84c876f4-2r472
22:58:41	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-runtime-predictor-6d84c876f4 from 0 to 1
22:58:41 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
22:58:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-2r472	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:58:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-2r472	Started	Started container storage-initializer
22:58:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-2r472	Created	Created container: storage-initializer
22:58:42	kserve-ci-e2e-test	multus	isvc-sklearn-v2-runtime-predictor-6d84c876f4-2r472	AddedInterface	Add eth0 [10.133.0.56/23] from ovn-kubernetes
22:58:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-2r472	Created	Created container: kserve-container
22:58:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-2r472	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
22:58:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-2r472	Started	Started container kube-rbac-proxy
22:58:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-2r472	Started	Started container kserve-container
22:58:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-2r472	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:58:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-2r472	Created	Created container: kube-rbac-proxy
22:59:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-2r472	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
22:59:26 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:59:26 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:59:33 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:59:33	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-v2-runtime] is Ready
22:59:41 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:59:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-2r472	Killing	Stopping container kserve-container
22:59:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-2r472	Killing	Stopping container kube-rbac-proxy
22:59:41	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-predictor-69755fbb9	SuccessfulCreate	Created pod: isvc-sklearn-v2-predictor-69755fbb9-kh7dl
22:59:41	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-predictor-69755fbb9 from 0 to 1
22:59:41 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:59:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
22:59:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-69755fbb9-kh7dl	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
22:59:42	kserve-ci-e2e-test	multus	isvc-sklearn-v2-predictor-69755fbb9-kh7dl	AddedInterface	Add eth0 [10.133.0.57/23] from ovn-kubernetes
22:59:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-69755fbb9-kh7dl	Created	Created container: storage-initializer
22:59:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-69755fbb9-kh7dl	Started	Started container storage-initializer
22:59:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-2r472	Unhealthy	Readiness probe failed: Get "http://10.133.0.56:8080/v2/models/isvc-sklearn-v2-runtime/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
22:59:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-69755fbb9-kh7dl	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1293" already present on machine
22:59:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-69755fbb9-kh7dl	Created	Created container: kserve-container
22:59:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-69755fbb9-kh7dl	Started	Started container kserve-container
22:59:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-69755fbb9-kh7dl	Started	Started container kube-rbac-proxy
22:59:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-69755fbb9-kh7dl	Created	Created container: kube-rbac-proxy
22:59:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-69755fbb9-kh7dl	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
22:59:48 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-2r472	Unhealthy	Readiness probe failed: Get "https://10.133.0.56:8643/healthz": dial tcp 10.133.0.56:8643: connect: connection refused
22:59:56	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:59:56	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:00:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	InferenceServiceReady	InferenceService [isvc-sklearn-v2] is Ready
23:00:53 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
23:00:56 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:00:56 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:01:02	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-mixed-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-mixed-predictor-7f8b779bc6 from 0 to 1
23:01:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7f8b779bc6-82fxz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
23:01:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-69755fbb9-kh7dl	Killing	Stopping container kube-rbac-proxy
23:01:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-69755fbb9-kh7dl	Killing	Stopping container kserve-container
23:01:02	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-mixed": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-mixed": the object has been modified; please apply your changes to the latest version and try again
23:01:02	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-mixed-predictor-7f8b779bc6	SuccessfulCreate	Created pod: isvc-sklearn-v2-mixed-predictor-7f8b779bc6-82fxz
23:01:02	kserve-ci-e2e-test	multus	isvc-sklearn-v2-mixed-predictor-7f8b779bc6-82fxz	AddedInterface	Add eth0 [10.133.0.58/23] from ovn-kubernetes
23:01:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7f8b779bc6-82fxz	Created	Created container: storage-initializer
23:01:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7f8b779bc6-82fxz	Started	Started container storage-initializer
23:01:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-69755fbb9-kh7dl	Unhealthy	Readiness probe failed: Get "https://10.133.0.57:8643/healthz": dial tcp 10.133.0.57:8643: connect: connection refused
23:01:03 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-69755fbb9-kh7dl	Unhealthy	Readiness probe failed: dial tcp 10.133.0.57:8080: connect: connection refused
23:01:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7f8b779bc6-82fxz	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1293" already present on machine
23:01:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7f8b779bc6-82fxz	Created	Created container: kserve-container
23:01:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7f8b779bc6-82fxz	Started	Started container kserve-container
23:01:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7f8b779bc6-82fxz	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
23:01:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7f8b779bc6-82fxz	Created	Created container: kube-rbac-proxy
23:01:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7f8b779bc6-82fxz	Started	Started container kube-rbac-proxy
23:01:47 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:01:47 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:02:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	InferenceServiceReady	InferenceService [isvc-sklearn-v2-mixed] is Ready
23:02:15 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
23:02:17 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:02:17 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:02:22	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-predictor-6756f669d7	SuccessfulCreate	Created pod: isvc-tensorflow-predictor-6756f669d7-gwwz8
23:02:22	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-gwwz8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
23:02:22	kserve-ci-e2e-test	multus	isvc-tensorflow-predictor-6756f669d7-gwwz8	AddedInterface	Add eth0 [10.133.0.59/23] from ovn-kubernetes
23:02:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7f8b779bc6-82fxz	Killing	Stopping container kserve-container
23:02:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7f8b779bc6-82fxz	Killing	Stopping container kube-rbac-proxy
23:02:22	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-predictor-6756f669d7 from 0 to 1
23:02:22	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-gwwz8	Created	Created container: storage-initializer
23:02:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow": the object has been modified; please apply your changes to the latest version and try again
23:02:22	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-gwwz8	Started	Started container storage-initializer
23:02:24 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7f8b779bc6-82fxz	Unhealthy	Readiness probe failed: dial tcp 10.133.0.58:8080: connect: connection refused
23:02:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7f8b779bc6-82fxz	Unhealthy	Readiness probe failed: Get "https://10.133.0.58:8643/healthz": dial tcp 10.133.0.58:8643: connect: connection refused
23:02:28	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-gwwz8	Pulling	Pulling image "tensorflow/serving:2.6.2"
23:02:31	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-gwwz8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
23:02:31	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-gwwz8	Created	Created container: kube-rbac-proxy
23:02:31	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-gwwz8	Started	Started container kube-rbac-proxy
23:02:31	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-gwwz8	Started	Started container kserve-container
23:02:31	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-gwwz8	Created	Created container: kserve-container
23:02:31	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-gwwz8	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 3.546s (3.546s including waiting). Image size: 425873876 bytes.
23:02:37	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:02:37	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:02:38 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-gwwz8	Unhealthy	Readiness probe failed: dial tcp 10.133.0.59:8080: connect: connection refused
23:02:48	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	InferenceServiceReady	InferenceService [isvc-tensorflow] is Ready
23:02:48 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
23:02:52	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:02:52	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:03:03	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-6kldd	Created	Created container: storage-initializer
23:03:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow-runtime": the object has been modified; please apply your changes to the latest version and try again
23:03:03	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-6kldd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
23:03:03	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-runtime-predictor-8699d78cf	SuccessfulCreate	Created pod: isvc-tensorflow-runtime-predictor-8699d78cf-6kldd
23:03:03	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-6kldd	Started	Started container storage-initializer
23:03:03	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-gwwz8	Killing	Stopping container kserve-container
23:03:03	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-gwwz8	Killing	Stopping container kube-rbac-proxy
23:03:03	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-runtime-predictor-8699d78cf from 0 to 1
23:03:03	kserve-ci-e2e-test	multus	isvc-tensorflow-runtime-predictor-8699d78cf-6kldd	AddedInterface	Add eth0 [10.133.0.60/23] from ovn-kubernetes
23:03:09	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-6kldd	Created	Created container: kserve-container
23:03:09	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-6kldd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
23:03:09	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-6kldd	Created	Created container: kube-rbac-proxy
23:03:09	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-6kldd	Started	Started container kserve-container
23:03:09	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-6kldd	Started	Started container kube-rbac-proxy
23:03:09	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-6kldd	Pulled	Container image "tensorflow/serving:2.6.2" already present on machine
23:03:17 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-6kldd	Unhealthy	Readiness probe failed: dial tcp 10.133.0.60:8080: connect: connection refused
23:03:18	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:03:18	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:03:27	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	InferenceServiceReady	InferenceService [isvc-tensorflow-runtime] is Ready
23:03:27 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
23:03:33	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:03:33 (x6)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-gwwz8	Unhealthy	Readiness probe failed: Get "https://10.133.0.59:8643/healthz": dial tcp 10.133.0.59:8643: connect: connection refused
23:03:33	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:03:44	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-nprnq	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-triton-predictor-serving-cert" not found
23:03:44	kserve-ci-e2e-test	deployment-controller	isvc-triton-predictor	ScalingReplicaSet	Scaled up replica set isvc-triton-predictor-84bb65d94b from 0 to 1
23:03:44	kserve-ci-e2e-test	replicaset-controller	isvc-triton-predictor-84bb65d94b	SuccessfulCreate	Created pod: isvc-triton-predictor-84bb65d94b-nprnq
23:03:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	UpdateFailed	Failed to update status for InferenceService "isvc-triton": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-triton": the object has been modified; please apply your changes to the latest version and try again
23:03:44	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-6kldd	Killing	Stopping container kserve-container
23:03:44	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-6kldd	Killing	Stopping container kube-rbac-proxy
23:03:45	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-nprnq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
23:03:45	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-nprnq	Started	Started container storage-initializer
23:03:45	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-nprnq	Created	Created container: storage-initializer
23:03:45	kserve-ci-e2e-test	multus	isvc-triton-predictor-84bb65d94b-nprnq	AddedInterface	Add eth0 [10.133.0.61/23] from ovn-kubernetes
23:03:49	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-nprnq	Pulling	Pulling image "nvcr.io/nvidia/tritonserver:23.05-py3"
23:04:12 (x6)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-6kldd	Unhealthy	Readiness probe failed: Get "https://10.133.0.60:8643/healthz": dial tcp 10.133.0.60:8643: connect: connection refused
23:05:43	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-nprnq	Started	Started container kserve-container
23:05:43	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-nprnq	Started	Started container kube-rbac-proxy
23:05:43	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-nprnq	Pulled	Successfully pulled image "nvcr.io/nvidia/tritonserver:23.05-py3" in 1m54.273s (1m54.273s including waiting). Image size: 12907074623 bytes.
23:05:43	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-nprnq	Created	Created container: kserve-container
23:05:43	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-nprnq	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
23:05:43	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-nprnq	Created	Created container: kube-rbac-proxy
23:05:44 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:05:44 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:05:45 (x2)	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-nprnq	Unhealthy	Readiness probe failed: dial tcp 10.133.0.61:8080: connect: connection refused
23:05:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	InferenceServiceReady	InferenceService [isvc-triton] is Ready
23:05:51 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
23:05:55	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-predictor-8689c4cfcc from 0 to 1
23:05:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost": the object has been modified; please apply your changes to the latest version and try again
23:05:55	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-nprnq	Unhealthy	Readiness probe failed: Get "https://10.133.0.61:8643/healthz": dial tcp 10.133.0.61:8643: connect: connection refused
23:05:55	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-nprnq	Killing	Stopping container kserve-container
23:05:55	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-predictor-8689c4cfcc	SuccessfulCreate	Created pod: isvc-xgboost-predictor-8689c4cfcc-67n8f
23:05:55	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-nprnq	Killing	Stopping container kube-rbac-proxy
23:05:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-67n8f	Created	Created container: storage-initializer
23:05:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-67n8f	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
23:05:56	kserve-ci-e2e-test	multus	isvc-xgboost-predictor-8689c4cfcc-67n8f	AddedInterface	Add eth0 [10.133.0.62/23] from ovn-kubernetes
23:05:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-67n8f	Started	Started container storage-initializer
23:06:00	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-67n8f	Pulling	Pulling image "kserve/xgbserver:latest"
23:06:19	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-67n8f	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 19.043s (19.043s including waiting). Image size: 1306229499 bytes.
23:06:20	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-67n8f	Started	Started container kube-rbac-proxy
23:06:20	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-67n8f	Created	Created container: kube-rbac-proxy
23:06:20	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-67n8f	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
23:06:20	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-67n8f	Started	Started container kserve-container
23:06:20	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-67n8f	Created	Created container: kserve-container
23:06:55 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:06:55 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:07:25 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:07:25 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:07:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	InferenceServiceReady	InferenceService [isvc-xgboost] is Ready
23:07:28 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
23:07:35	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-67n8f	Killing	Stopping container kube-rbac-proxy
23:07:35	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-67n8f	Killing	Stopping container kserve-container
23:07:36	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-mlserver-predictor-67d4bc6646	SuccessfulCreate	Created pod: isvc-xgboost-v2-mlserver-predictor-67d4bc6646-rrdgw
23:07:36	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-mlserver-predictor-67d4bc6646 from 0 to 1
23:07:36	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-rrdgw	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
23:07:36	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
23:07:36 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
23:07:37	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-rrdgw	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
23:07:37	kserve-ci-e2e-test	multus	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-rrdgw	AddedInterface	Add eth0 [10.133.0.63/23] from ovn-kubernetes
23:07:37	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-rrdgw	Started	Started container storage-initializer
23:07:37	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-rrdgw	Created	Created container: storage-initializer
23:07:38	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-67n8f	Unhealthy	Readiness probe failed: Get "https://10.133.0.62:8643/healthz": dial tcp 10.133.0.62:8643: connect: connection refused
23:07:38 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-67n8f	Unhealthy	Readiness probe failed: dial tcp 10.133.0.62:8080: connect: connection refused
23:07:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-rrdgw	Started	Started container kube-rbac-proxy
23:07:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-rrdgw	Started	Started container kserve-container
23:07:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-rrdgw	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
23:07:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-rrdgw	Created	Created container: kube-rbac-proxy
23:07:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-rrdgw	Created	Created container: kserve-container
23:07:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-rrdgw	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
23:07:51	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:07:51	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:08:19 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
23:08:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	InferenceServiceReady	InferenceService [isvc-xgboost-v2-mlserver] is Ready
23:08:21 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:08:21 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:08:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-rrdgw	Killing	Stopping container kube-rbac-proxy
23:08:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-rrdgw	Killing	Stopping container kserve-container
23:08:26	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-5ftr9	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "xgboost-v2-mlserver-predictor-serving-cert" not found
23:08:26	kserve-ci-e2e-test	replicaset-controller	xgboost-v2-mlserver-predictor-7799869d6f	SuccessfulCreate	Created pod: xgboost-v2-mlserver-predictor-7799869d6f-5ftr9
23:08:26	kserve-ci-e2e-test	deployment-controller	xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set xgboost-v2-mlserver-predictor-7799869d6f from 0 to 1
23:08:26	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
23:08:27	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-5ftr9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
23:08:27	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-5ftr9	Started	Started container storage-initializer
23:08:27	kserve-ci-e2e-test	multus	xgboost-v2-mlserver-predictor-7799869d6f-5ftr9	AddedInterface	Add eth0 [10.133.0.64/23] from ovn-kubernetes
23:08:27	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-5ftr9	Created	Created container: storage-initializer
23:08:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-rrdgw	Unhealthy	Readiness probe failed: Get "https://10.133.0.63:8643/healthz": dial tcp 10.133.0.63:8643: connect: connection refused
23:08:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-rrdgw	Unhealthy	Readiness probe failed: Get "http://10.133.0.63:8080/v2/models/isvc-xgboost-v2-mlserver/ready": dial tcp 10.133.0.63:8080: connect: connection refused
23:08:31	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-5ftr9	Started	Started container kserve-container
23:08:31	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-5ftr9	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
23:08:31	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-5ftr9	Created	Created container: kserve-container
23:08:31	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-5ftr9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
23:08:31	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-5ftr9	Created	Created container: kube-rbac-proxy
23:08:31	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-5ftr9	Started	Started container kube-rbac-proxy
23:08:56	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:08:56	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:09:08	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	InferenceServiceReady	InferenceService [xgboost-v2-mlserver] is Ready
23:09:08 (x12)	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
23:09:11 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:09:11 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:09:16	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-5ftr9	Killing	Stopping container kube-rbac-proxy
23:09:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-runtime": the object has been modified; please apply your changes to the latest version and try again
23:09:16	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-5ftr9	Killing	Stopping container kserve-container
23:09:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-z2km9	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
23:09:16	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-runtime-predictor-779db84d9	SuccessfulCreate	Created pod: isvc-xgboost-runtime-predictor-779db84d9-z2km9
23:09:16	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-runtime-predictor-779db84d9 from 0 to 1
23:09:17	kserve-ci-e2e-test	multus	isvc-xgboost-runtime-predictor-779db84d9-z2km9	AddedInterface	Add eth0 [10.133.0.65/23] from ovn-kubernetes
23:09:17	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-z2km9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
23:09:17	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-z2km9	Created	Created container: storage-initializer
23:09:17	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-z2km9	Started	Started container storage-initializer
23:09:18	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-5ftr9	Unhealthy	Readiness probe failed: Get "https://10.133.0.64:8643/healthz": dial tcp 10.133.0.64:8643: connect: connection refused
23:09:21	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-z2km9	Created	Created container: kserve-container
23:09:21	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-z2km9	Pulled	Container image "kserve/xgbserver:latest" already present on machine
23:09:21	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-z2km9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
23:09:21	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-z2km9	Created	Created container: kube-rbac-proxy
23:09:21	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-z2km9	Started	Started container kube-rbac-proxy
23:09:21	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-z2km9	Started	Started container kserve-container
23:10:01 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:10:01 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:10:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-runtime] is Ready
23:10:28 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
23:10:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:10:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:10:36	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-runtime-predictor-6dc5954dc from 0 to 1
23:10:36	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-z2km9	Killing	Stopping container kube-rbac-proxy
23:10:36	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-z2km9	Killing	Stopping container kserve-container
23:10:36	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-xhb2s	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
23:10:36	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
23:10:36	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-runtime-predictor-6dc5954dc	SuccessfulCreate	Created pod: isvc-xgboost-v2-runtime-predictor-6dc5954dc-xhb2s
23:10:37	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-xhb2s	Started	Started container storage-initializer
23:10:37	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-xhb2s	Created	Created container: storage-initializer
23:10:37	kserve-ci-e2e-test	multus	isvc-xgboost-v2-runtime-predictor-6dc5954dc-xhb2s	AddedInterface	Add eth0 [10.133.0.66/23] from ovn-kubernetes
23:10:37	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-xhb2s	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
23:10:38 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-z2km9	Unhealthy	Readiness probe failed: dial tcp 10.133.0.65:8080: connect: connection refused
23:10:38	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-z2km9	Unhealthy	Readiness probe failed: Get "https://10.133.0.65:8643/healthz": dial tcp 10.133.0.65:8643: connect: connection refused
23:10:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-xhb2s	Started	Started container kserve-container
23:10:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-xhb2s	Created	Created container: kserve-container
23:10:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-xhb2s	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
23:10:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-xhb2s	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
23:10:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-xhb2s	Created	Created container: kube-rbac-proxy
23:10:42	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-xhb2s	Started	Started container kube-rbac-proxy
23:10:51	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:10:51	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:11:18	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-xhb2s	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
23:11:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-v2-runtime] is Ready
23:11:28 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
23:11:36 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:11:36	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
23:11:36	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-xctlh	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-v2-predictor-serving-cert" not found
23:11:36 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:11:36	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-predictor-6fcdd6977c	SuccessfulCreate	Created pod: isvc-xgboost-v2-predictor-6fcdd6977c-xctlh
23:11:36	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-predictor-6fcdd6977c from 0 to 1
23:11:36	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-xhb2s	Killing	Stopping container kserve-container
23:11:36	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-xhb2s	Killing	Stopping container kube-rbac-proxy
23:11:37	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-xctlh	Created	Created container: storage-initializer
23:11:37	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-xctlh	Started	Started container storage-initializer
23:11:37	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-xctlh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
23:11:37	kserve-ci-e2e-test	multus	isvc-xgboost-v2-predictor-6fcdd6977c-xctlh	AddedInterface	Add eth0 [10.133.0.67/23] from ovn-kubernetes
23:11:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-xhb2s	Unhealthy	Readiness probe failed: Get "http://10.133.0.66:8080/v2/models/isvc-xgboost-v2-runtime/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
23:11:42	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-xctlh	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
23:11:42	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-xctlh	Started	Started container kube-rbac-proxy
23:11:42	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-xctlh	Created	Created container: kube-rbac-proxy
23:11:42	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-xctlh	Started	Started container kserve-container
23:11:42	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-xctlh	Created	Created container: kserve-container
23:11:42	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-xctlh	Pulled	Container image "kserve/xgbserver:latest" already present on machine
23:11:43 (x2)	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-xhb2s	Unhealthy	Readiness probe failed: Get "https://10.133.0.66:8643/healthz": dial tcp 10.133.0.66:8643: connect: connection refused
23:12:06 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:12:06 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:12:50 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
23:12:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	InferenceServiceReady	InferenceService [isvc-xgboost-v2] is Ready
23:12:51 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:12:51 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:12:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-xctlh	Killing	Stopping container kube-rbac-proxy
23:12:56	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-xctlh	Killing	Stopping container kserve-container
23:12:56	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-predictor-88457d696 from 0 to 1
23:12:56	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3": the object has been modified; please apply your changes to the latest version and try again
23:12:56	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-predictor-88457d696	SuccessfulCreate	Created pod: isvc-sklearn-s3-predictor-88457d696-6fqtp
23:12:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-88457d696-6fqtp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
23:12:57	kserve-ci-e2e-test	multus	isvc-sklearn-s3-predictor-88457d696-6fqtp	AddedInterface	Add eth0 [10.133.0.68/23] from ovn-kubernetes
23:12:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-88457d696-6fqtp	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-predictor-serving-cert" not found
23:12:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-88457d696-6fqtp	Started	Started container storage-initializer
23:12:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-88457d696-6fqtp	Created	Created container: storage-initializer
23:12:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-88457d696-6fqtp	Started	Started container kube-rbac-proxy
23:12:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-88457d696-6fqtp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
23:12:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-88457d696-6fqtp	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1293" already present on machine
23:12:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-88457d696-6fqtp	Created	Created container: kserve-container
23:12:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-88457d696-6fqtp	Started	Started container kserve-container
23:12:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-88457d696-6fqtp	Created	Created container: kube-rbac-proxy
23:13:00	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-xctlh	Unhealthy	Readiness probe failed: Get "https://10.133.0.67:8643/healthz": dial tcp 10.133.0.67:8643: connect: connection refused
23:13:00 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-xctlh	Unhealthy	Readiness probe failed: dial tcp 10.133.0.67:8080: connect: connection refused
23:13:11	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:13:11	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:14:07 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
23:14:07	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	InferenceServiceReady	InferenceService [isvc-sklearn-s3] is Ready
23:14:11 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:14:11 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:14:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-rkvjg	Started	Started container storage-initializer
23:14:17	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-rkvjg	AddedInterface	Add eth0 [10.133.0.69/23] from ovn-kubernetes
23:14:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-88457d696-6fqtp	Unhealthy	Readiness probe failed: Get "https://10.133.0.68:8643/healthz": dial tcp 10.133.0.68:8643: connect: connection refused
23:14:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-88457d696-6fqtp	Killing	Stopping container kube-rbac-proxy
23:14:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-88457d696-6fqtp	Killing	Stopping container kserve-container
23:14:17 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-88457d696-6fqtp	Unhealthy	Readiness probe failed: dial tcp 10.133.0.68:8080: connect: connection refused
23:14:17	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-pass-predictor-5488974f76	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-rkvjg
23:14:17	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-pass-predictor-5488974f76 from 0 to 1
23:14:17	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-pass": the object has been modified; please apply your changes to the latest version and try again
23:14:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-rkvjg	Created	Created container: storage-initializer
23:14:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-rkvjg	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
23:14:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-rkvjg	Started	Started container kserve-container
23:14:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-rkvjg	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1293" already present on machine
23:14:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-rkvjg	Created	Created container: kube-rbac-proxy
23:14:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-rkvjg	Started	Started container kube-rbac-proxy
23:14:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-rkvjg	Created	Created container: kserve-container
23:14:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-rkvjg	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
23:15:02 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:15:02 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:15:27	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-global-pass] is Ready
23:15:27 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
23:15:32 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:15:32 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:15:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-rkvjg	Killing	Stopping container kube-rbac-proxy
23:15:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-rkvjg	Unhealthy	Readiness probe failed: Get "https://10.133.0.69:8643/healthz": dial tcp 10.133.0.69:8643: connect: connection refused
23:15:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-rkvjg	Killing	Stopping container kserve-container
23:15:37 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-5488974f76-rkvjg	Unhealthy	Readiness probe failed: dial tcp 10.133.0.69:8080: connect: connection refused
23:15:38	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-fail-predictor-54884788bb from 0 to 1
23:15:38	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-fail-predictor-54884788bb	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-7fb4l
23:15:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-fail": the object has been modified; please apply your changes to the latest version and try again
23:15:38	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-7fb4l	AddedInterface	Add eth0 [10.133.0.70/23] from ovn-kubernetes
23:15:41 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-7fb4l	Started	Started container storage-initializer
23:15:41 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-7fb4l	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
23:15:41 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-7fb4l	Created	Created container: storage-initializer
23:15:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-7fb4l	BackOff	Back-off restarting failed container storage-initializer in pod isvc-sklearn-s3-tls-global-fail-predictor-54884788bb-7fb4l_kserve-ci-e2e-test(a9f060d0-fa40-44e4-8f4a-dd989af97836)
23:15:46 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
23:15:49	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664 from 0 to 1
23:15:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-pass": the object has been modified; please apply your changes to the latest version and try again
23:15:49	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-nmm2l
23:15:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-nmm2l	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
23:15:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-nmm2l	Created	Created container: storage-initializer
23:15:50	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-nmm2l	AddedInterface	Add eth0 [10.133.0.71/23] from ovn-kubernetes
23:15:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-nmm2l	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
23:15:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-nmm2l	Started	Started container storage-initializer
23:15:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-nmm2l	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
23:15:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-nmm2l	Created	Created container: kserve-container
23:15:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-nmm2l	Started	Started container kserve-container
23:15:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-nmm2l	Created	Created container: kube-rbac-proxy
23:15:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-nmm2l	Started	Started container kube-rbac-proxy
23:15:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-nmm2l	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1293" already present on machine
23:16:04	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:16:04	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:16:59	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-custom-pass] is Ready
23:16:59 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
23:17:04 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:17:04 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:17:09 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-nmm2l	Unhealthy	Readiness probe failed: dial tcp 10.133.0.71:8080: connect: connection refused
23:17:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-nmm2l	Unhealthy	Readiness probe failed: Get "https://10.133.0.71:8643/healthz": dial tcp 10.133.0.71:8643: connect: connection refused
23:17:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-nmm2l	Killing	Stopping container kube-rbac-proxy
23:17:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-7877ccc664-nmm2l	Killing	Stopping container kserve-container
23:17:10	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd from 0 to 1
23:17:10	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-pqvbj	AddedInterface	Add eth0 [10.133.0.72/23] from ovn-kubernetes
23:17:10	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-pqvbj
23:17:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-fail": the object has been modified; please apply your changes to the latest version and try again
23:17:15 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-pqvbj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
23:17:15 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-pqvbj	Created	Created container: storage-initializer
23:17:15 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-pqvbj	Started	Started container storage-initializer
23:17:15 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
23:17:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-7d65b5b7cd-pqvbj	Killing	Stopping container storage-initializer
23:17:21	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf from 0 to 1
23:17:21	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-n2hcp
23:17:21	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-pass": the object has been modified; please apply your changes to the latest version and try again
23:17:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-n2hcp	Started	Started container storage-initializer
23:17:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-n2hcp	Created	Created container: storage-initializer
23:17:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-n2hcp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
23:17:21	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-n2hcp	AddedInterface	Add eth0 [10.133.0.73/23] from ovn-kubernetes
23:17:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-n2hcp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
23:17:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-n2hcp	Created	Created container: kube-rbac-proxy
23:17:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-n2hcp	Started	Started container kube-rbac-proxy
23:17:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-n2hcp	Started	Started container kserve-container
23:17:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-n2hcp	Created	Created container: kserve-container
23:17:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-n2hcp	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1293" already present on machine
23:17:36	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:17:36	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
23:18:20 (x8)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-n2hcp	Unhealthy	Readiness probe failed: dial tcp 10.133.0.73:8080: connect: connection refused
23:18:21 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:18:21 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
23:18:30 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
23:18:30	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-serving-pass] is Ready
23:18:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-n2hcp	Killing	Stopping container kserve-container
23:18:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-n2hcp	Killing	Stopping container kube-rbac-proxy
23:18:32	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-8r5vr	AddedInterface	Add eth0 [10.133.0.74/23] from ovn-kubernetes
23:18:32	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-fail": the object has been modified; please apply your changes to the latest version and try again
23:18:32	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965 from 0 to 1
23:18:32	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-8r5vr
23:18:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-c86b5bbcf-n2hcp	Unhealthy	Readiness probe failed: Get "https://10.133.0.73:8643/healthz": dial tcp 10.133.0.73:8643: connect: connection refused
23:18:39 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
23:18:39 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-8r5vr	Started	Started container storage-initializer
23:18:39 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-8r5vr	Created	Created container: storage-initializer
23:18:39 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-8r5vr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:43c598108b2566718d0b8a06fae8e38f22d997b9ce2bbd3c912a448d6236ab6b" already present on machine
23:18:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-5bc5655965-8r5vr	Killing	Stopping container storage-initializer