Events

Time	Namespace	Component	RelatedObject	Reason	Message
16:50:51	kserve-ci-e2e-test		isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72 to ip-10-0-130-83.ec2.internal
17:04:07	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm to ip-10-0-130-83.ec2.internal
16:46:53	kserve-ci-e2e-test		sklearn-v2-mlserver-predictor-65d8664766-llw97	Scheduled	Successfully assigned kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97 to ip-10-0-130-83.ec2.internal
16:15:48	kserve-ci-e2e-test		isvc-logger-predictor-6d94fc7bdd-jxjrv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv to ip-10-0-130-83.ec2.internal
16:24:44	kserve-ci-e2e-test		isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92 to ip-10-0-130-83.ec2.internal
16:35:59	kserve-ci-e2e-test		isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4 to ip-10-0-130-83.ec2.internal
16:28:18	kserve-ci-e2e-test		isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z to ip-10-0-130-83.ec2.internal
16:36:09	kserve-ci-e2e-test		isvc-predictive-sklearn-predictor-cd7c759c9-kc74j	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j to ip-10-0-130-83.ec2.internal
16:45:33	kserve-ci-e2e-test		isvc-sklearn-predictor-bc9cf9b57-k9h2x	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x to ip-10-0-130-83.ec2.internal
16:32:42	kserve-ci-e2e-test		isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz to ip-10-0-130-83.ec2.internal
16:15:28	kserve-ci-e2e-test		message-dumper-predictor-c7d86bcbd-7bgsr	Scheduled	Successfully assigned kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr to ip-10-0-130-83.ec2.internal
16:17:13	kserve-ci-e2e-test		isvc-lightgbm-predictor-bdf964bd-r64vj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj to ip-10-0-130-83.ec2.internal
17:00:26	kserve-ci-e2e-test		isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4 to ip-10-0-130-83.ec2.internal
16:42:31	kserve-ci-e2e-test		isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n to ip-10-0-130-83.ec2.internal
17:01:26	kserve-ci-e2e-test		isvc-xgboost-v2-predictor-6fcdd6977c-78tfs	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs to ip-10-0-130-83.ec2.internal
16:57:26	kserve-ci-e2e-test		isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd to ip-10-0-130-83.ec2.internal
16:59:06	kserve-ci-e2e-test		isvc-xgboost-runtime-predictor-779db84d9-fmnv4	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4 to ip-10-0-130-83.ec2.internal
16:55:45	kserve-ci-e2e-test		isvc-xgboost-predictor-8689c4cfcc-rtqwj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj to ip-10-0-130-83.ec2.internal
16:14:02	kserve-ci-e2e-test		isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk to ip-10-0-130-83.ec2.internal
16:35:43	kserve-ci-e2e-test		isvc-secondary-75c1b6-predictor-cdb677c59-skj8w	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w to ip-10-0-130-83.ec2.internal
16:34:23	kserve-ci-e2e-test		isvc-primary-75c1b6-predictor-57dff89c5b-9gt85	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85 to ip-10-0-130-83.ec2.internal
16:38:00	kserve-ci-e2e-test		isvc-predictive-xgboost-predictor-7ff98fd74d-868qv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv to ip-10-0-130-83.ec2.internal
16:31:01	kserve-ci-e2e-test		isvc-pmml-runtime-predictor-67bc544947-lhzhb	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb to ip-10-0-130-83.ec2.internal
16:20:14	kserve-ci-e2e-test		isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp to ip-10-0-130-83.ec2.internal
16:53:34	kserve-ci-e2e-test		isvc-triton-predictor-84bb65d94b-wg8gk	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk to ip-10-0-130-83.ec2.internal
16:52:52	kserve-ci-e2e-test		isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl to ip-10-0-130-83.ec2.internal
16:52:11	kserve-ci-e2e-test		isvc-tensorflow-predictor-6756f669d7-78wr2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2 to ip-10-0-130-83.ec2.internal
16:48:30	kserve-ci-e2e-test		isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf to ip-10-0-130-83.ec2.internal
16:49:31	kserve-ci-e2e-test		isvc-sklearn-v2-predictor-9df9c96c7-862j2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2 to ip-10-0-130-83.ec2.internal
16:27:16	kserve-ci-e2e-test		isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm to ip-10-0-130-83.ec2.internal
16:58:16	kserve-ci-e2e-test		xgboost-v2-mlserver-predictor-7799869d6f-ppp6x	Scheduled	Successfully assigned kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x to ip-10-0-130-83.ec2.internal
16:44:01	kserve-ci-e2e-test		isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx to ip-10-0-130-83.ec2.internal
16:26:04	kserve-ci-e2e-test		isvc-paddle-predictor-6b8b7cfb4b-r58lh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh to ip-10-0-130-83.ec2.internal
16:29:19	kserve-ci-e2e-test		isvc-pmml-predictor-8bb578669-4mm66	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66 to ip-10-0-130-83.ec2.internal
16:18:43	kserve-ci-e2e-test		isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt to ip-10-0-130-83.ec2.internal
16:41:00	kserve-ci-e2e-test		isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm to ip-10-0-130-83.ec2.internal
17:07:11	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v to ip-10-0-130-83.ec2.internal
17:05:28	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db to ip-10-0-130-83.ec2.internal
17:05:39	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq to ip-10-0-130-83.ec2.internal
17:07:00	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284 to ip-10-0-130-83.ec2.internal
17:02:46	kserve-ci-e2e-test		isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5 to ip-10-0-130-83.ec2.internal
16:47:53	kserve-ci-e2e-test		isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl to ip-10-0-130-83.ec2.internal
16:12:17	kserve-ci-e2e-test		isvc-sklearn-batcher-predictor-54b87fc785-r767m	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m to ip-10-0-130-83.ec2.internal
16:23:24	kserve-ci-e2e-test		isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv to ip-10-0-130-83.ec2.internal
16:39:30	kserve-ci-e2e-test		isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz to ip-10-0-130-83.ec2.internal
16:23:54	kserve-ci-e2e-test		isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq to ip-10-0-130-83.ec2.internal
17:08:32	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8 to ip-10-0-130-83.ec2.internal
16:12:17	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
16:12:17	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-predictor-54b87fc785	SuccessfulCreate	Created pod: isvc-sklearn-batcher-predictor-54b87fc785-r767m
16:12:17	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
16:12:17	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-predictor-54b87fc785 from 0 to 1
16:12:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-54b87fc785-r767m	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
16:12:18	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-predictor-54b87fc785-r767m	AddedInterface	Add eth0 [10.132.0.21/23] from ovn-kubernetes
16:12:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-54b87fc785-r767m	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341"
16:12:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-54b87fc785-r767m	Created	Created container: storage-initializer
16:12:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-54b87fc785-r767m	Started	Started container storage-initializer
16:12:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-54b87fc785-r767m	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" in 3.641s (3.641s including waiting). Image size: 301466548 bytes.
16:12:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-54b87fc785-r767m	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1532"
16:12:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-54b87fc785-r767m	Started	Started container kserve-container
16:12:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-54b87fc785-r767m	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
16:12:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-54b87fc785-r767m	Created	Created container: kserve-container
16:12:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-54b87fc785-r767m	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1532" in 13.163s (13.163s including waiting). Image size: 1560933300 bytes.
16:12:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-54b87fc785-r767m	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.372s (2.372s including waiting). Image size: 211946088 bytes.
16:12:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-54b87fc785-r767m	Created	Created container: kube-rbac-proxy
16:12:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-54b87fc785-r767m	Started	Started container kube-rbac-proxy
16:12:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-54b87fc785-r767m	Pulling	Pulling image "quay.io/opendatahub/kserve-agent@sha256:d4dfc83be5dd58d8739a157278436e5a923b7626716bcc5e76e3c331a25671b6"
16:12:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-54b87fc785-r767m	Started	Started container agent
16:12:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-54b87fc785-r767m	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:d4dfc83be5dd58d8739a157278436e5a923b7626716bcc5e76e3c331a25671b6" in 2.186s (2.186s including waiting). Image size: 239690326 bytes.
16:12:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-54b87fc785-r767m	Created	Created container: agent
16:13:17 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:13:17 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:13:47 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:13:47 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:13:48	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InferenceServiceReady	InferenceService [isvc-sklearn-batcher] is Ready
16:13:48 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:14:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-54b87fc785-r767m	Killing	Stopping container kube-rbac-proxy
16:14:02	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-custom-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-custom-predictor-755c4698cb from 0 to 1
16:14:02	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-custom-predictor-755c4698cb	SuccessfulCreate	Created pod: isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk
16:14:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-54b87fc785-r767m	Killing	Stopping container kserve-container
16:14:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-54b87fc785-r767m	Killing	Stopping container agent
16:14:02 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher-custom": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher-custom": the object has been modified; please apply your changes to the latest version and try again
16:14:02	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher-custom": the object has been modified; please apply your changes to the latest version and try again
16:14:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:14:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk	Created	Created container: storage-initializer
16:14:03	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk	AddedInterface	Add eth0 [10.132.0.22/23] from ovn-kubernetes
16:14:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk	Started	Started container storage-initializer
16:14:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
16:14:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:14:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1532" already present on machine
16:14:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk	Started	Started container kserve-container
16:14:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk	Started	Started container kube-rbac-proxy
16:14:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:d4dfc83be5dd58d8739a157278436e5a923b7626716bcc5e76e3c331a25671b6" already present on machine
16:14:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk	Started	Started container agent
16:14:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk	Created	Created container: agent
16:14:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk	Created	Created container: kube-rbac-proxy
16:14:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk	Created	Created container: kserve-container
16:14:17 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-54b87fc785-r767m	Unhealthy	Readiness probe failed: Get "https://10.132.0.21:8643/healthz": dial tcp 10.132.0.21:8643: connect: connection refused
16:14:18 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-54b87fc785-r767m	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
16:14:18 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-54b87fc785-r767m	Unhealthy	Readiness probe failed: dial tcp 10.132.0.21:8080: connect: connection refused
16:14:32 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:14:32 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:15:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	InferenceServiceReady	InferenceService [isvc-sklearn-batcher-custom] is Ready
16:15:16 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:15:17 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:15:17 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:15:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk	Killing	Stopping container kserve-container
16:15:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk	Killing	Stopping container agent
16:15:27	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	UpdateFailed	Failed to update status for InferenceService "message-dumper": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper": the object has been modified; please apply your changes to the latest version and try again
16:15:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk	Killing	Stopping container kube-rbac-proxy
16:15:27	kserve-ci-e2e-test	deployment-controller	message-dumper-predictor	ScalingReplicaSet	Scaled up replica set message-dumper-predictor-c7d86bcbd from 0 to 1
16:15:28	kserve-ci-e2e-test	replicaset-controller	message-dumper-predictor-c7d86bcbd	SuccessfulCreate	Created pod: message-dumper-predictor-c7d86bcbd-7bgsr
16:15:28	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-7bgsr	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "message-dumper-predictor-serving-cert" not found
16:15:29	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-7bgsr	Pulled	Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 882ms (882ms including waiting). Image size: 14813193 bytes.
16:15:29	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-7bgsr	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:15:29	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-7bgsr	Started	Started container kserve-container
16:15:29	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-7bgsr	Created	Created container: kserve-container
16:15:29	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-7bgsr	Pulling	Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display"
16:15:29	kserve-ci-e2e-test	multus	message-dumper-predictor-c7d86bcbd-7bgsr	AddedInterface	Add eth0 [10.132.0.23/23] from ovn-kubernetes
16:15:30	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-7bgsr	Started	Started container kube-rbac-proxy
16:15:30	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-7bgsr	Created	Created container: kube-rbac-proxy
16:15:36 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
16:15:38	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	InferenceServiceReady	InferenceService [message-dumper] is Ready
16:15:38 (x8)	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:15:42	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:15:42	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:15:46 (x4)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk	Unhealthy	Readiness probe failed: Get "https://10.132.0.22:8643/healthz": dial tcp 10.132.0.22:8643: connect: connection refused
16:15:46 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk	Unhealthy	Readiness probe failed: dial tcp 10.132.0.22:5000: connect: connection refused
16:15:48	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6d94fc7bdd-jxjrv	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-logger-predictor-serving-cert" not found
16:15:48	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	UpdateFailed	Failed to update status for InferenceService "isvc-logger": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger": the object has been modified; please apply your changes to the latest version and try again
16:15:48	kserve-ci-e2e-test	deployment-controller	isvc-logger-predictor	ScalingReplicaSet	Scaled up replica set isvc-logger-predictor-6d94fc7bdd from 0 to 1
16:15:48	kserve-ci-e2e-test	replicaset-controller	isvc-logger-predictor-6d94fc7bdd	SuccessfulCreate	Created pod: isvc-logger-predictor-6d94fc7bdd-jxjrv
16:15:49	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6d94fc7bdd-jxjrv	Started	Started container storage-initializer
16:15:49	kserve-ci-e2e-test	multus	isvc-logger-predictor-6d94fc7bdd-jxjrv	AddedInterface	Add eth0 [10.132.0.24/23] from ovn-kubernetes
16:15:49	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6d94fc7bdd-jxjrv	Created	Created container: storage-initializer
16:15:49	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6d94fc7bdd-jxjrv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:15:53	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6d94fc7bdd-jxjrv	Created	Created container: kserve-container
16:15:53	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6d94fc7bdd-jxjrv	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1532" already present on machine
16:15:53	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6d94fc7bdd-jxjrv	Started	Started container kserve-container
16:15:53	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6d94fc7bdd-jxjrv	Created	Created container: agent
16:15:53	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6d94fc7bdd-jxjrv	Started	Started container agent
16:15:53	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6d94fc7bdd-jxjrv	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:d4dfc83be5dd58d8739a157278436e5a923b7626716bcc5e76e3c331a25671b6" already present on machine
16:15:53	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6d94fc7bdd-jxjrv	Started	Started container kube-rbac-proxy
16:15:53	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6d94fc7bdd-jxjrv	Created	Created container: kube-rbac-proxy
16:15:53	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6d94fc7bdd-jxjrv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:16:13 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:16:13 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:16:18 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:16:18 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:17:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	InferenceServiceReady	InferenceService [isvc-logger] is Ready
16:17:00 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:17:03 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:17:03 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:17:13	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-predictor-bdf964bd	SuccessfulCreate	Created pod: isvc-lightgbm-predictor-bdf964bd-r64vj
16:17:13	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-7bgsr	Killing	Stopping container kserve-container
16:17:13	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-predictor-bdf964bd from 0 to 1
16:17:13	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-7bgsr	Unhealthy	Readiness probe failed: Get "https://10.132.0.23:8643/healthz": dial tcp 10.132.0.23:8643: connect: connection refused
16:17:13	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm": the object has been modified; please apply your changes to the latest version and try again
16:17:13	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6d94fc7bdd-jxjrv	Killing	Stopping container kserve-container
16:17:13	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6d94fc7bdd-jxjrv	Killing	Stopping container agent
16:17:13	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6d94fc7bdd-jxjrv	Killing	Stopping container kube-rbac-proxy
16:17:13	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-7bgsr	Killing	Stopping container kube-rbac-proxy
16:17:13	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-r64vj	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-lightgbm-predictor-serving-cert" not found
16:17:14	kserve-ci-e2e-test	multus	isvc-lightgbm-predictor-bdf964bd-r64vj	AddedInterface	Add eth0 [10.132.0.25/23] from ovn-kubernetes
16:17:14	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-r64vj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:17:14	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-r64vj	Created	Created container: storage-initializer
16:17:14	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-r64vj	Started	Started container storage-initializer
16:17:18	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-r64vj	Pulling	Pulling image "kserve/lgbserver:latest"
16:17:25	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-r64vj	Pulled	Successfully pulled image "kserve/lgbserver:latest" in 6.331s (6.331s including waiting). Image size: 606724367 bytes.
16:17:25	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-r64vj	Started	Started container kube-rbac-proxy
16:17:25	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-r64vj	Created	Created container: kserve-container
16:17:25	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-r64vj	Started	Started container kserve-container
16:17:25	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-r64vj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:17:25	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-r64vj	Created	Created container: kube-rbac-proxy
16:17:30 (x4)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6d94fc7bdd-jxjrv	Unhealthy	Readiness probe failed: Get "https://10.132.0.24:8643/healthz": dial tcp 10.132.0.24:8643: connect: connection refused
16:17:30 (x10)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6d94fc7bdd-jxjrv	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
16:17:30 (x10)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6d94fc7bdd-jxjrv	Unhealthy	Readiness probe failed: dial tcp 10.132.0.24:8080: connect: connection refused
16:18:13 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:18:13 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:18:32 (x9)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-r64vj	Unhealthy	Readiness probe failed: dial tcp 10.132.0.25:8080: connect: connection refused
16:18:42 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:18:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	InferenceServiceReady	InferenceService [isvc-lightgbm] is Ready
16:18:43	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-runtime-predictor-749c4f6d58	SuccessfulCreate	Created pod: isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt
16:18:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:18:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:18:43	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
16:18:43	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-r64vj	Killing	Stopping container kube-rbac-proxy
16:18:43	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-r64vj	Killing	Stopping container kserve-container
16:18:43	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-runtime": the object has been modified; please apply your changes to the latest version and try again
16:18:43	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-runtime-predictor-749c4f6d58 from 0 to 1
16:18:44	kserve-ci-e2e-test	multus	isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt	AddedInterface	Add eth0 [10.132.0.26/23] from ovn-kubernetes
16:18:44	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:18:44	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt	Created	Created container: storage-initializer
16:18:44	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt	Started	Started container storage-initializer
16:18:47	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-r64vj	Unhealthy	Readiness probe failed: Get "https://10.132.0.25:8643/healthz": dial tcp 10.132.0.25:8643: connect: connection refused
16:18:49	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt	Created	Created container: kube-rbac-proxy
16:18:49	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt	Started	Started container kserve-container
16:18:49	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt	Started	Started container kube-rbac-proxy
16:18:49	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt	Pulled	Container image "kserve/lgbserver:latest" already present on machine
16:18:49	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt	Created	Created container: kserve-container
16:18:49	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:19:13 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:19:13 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:20:07 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:20:07	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-runtime] is Ready
16:20:13 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:20:13 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:20:14	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt	Killing	Stopping container kube-rbac-proxy
16:20:14	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt	Killing	Stopping container kserve-container
16:20:14	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
16:20:14	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-runtime-predictor-8765c9667	SuccessfulCreate	Created pod: isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp
16:20:14	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-runtime-predictor-8765c9667 from 0 to 1
16:20:14 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
16:20:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
16:20:15	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:20:15	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp	Started	Started container storage-initializer
16:20:15	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp	Created	Created container: storage-initializer
16:20:15	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp	AddedInterface	Add eth0 [10.132.0.27/23] from ovn-kubernetes
16:20:17 (x10)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt	Unhealthy	Readiness probe failed: dial tcp 10.132.0.26:8080: connect: connection refused
16:20:17	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt	Unhealthy	Readiness probe failed: Get "https://10.132.0.26:8643/healthz": dial tcp 10.132.0.26:8643: connect: connection refused
16:20:19	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp	Pulling	Pulling image "docker.io/seldonio/mlserver:1.7.1"
16:22:37	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp	Started	Started container kube-rbac-proxy
16:22:37	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp	Pulled	Successfully pulled image "docker.io/seldonio/mlserver:1.7.1" in 2m17.685s (2m17.685s including waiting). Image size: 10890461297 bytes.
16:22:37	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp	Created	Created container: kube-rbac-proxy
16:22:37	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp	Created	Created container: kserve-container
16:22:37	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:22:37	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp	Started	Started container kserve-container
16:22:44 (x10)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:22:44 (x10)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:23:14 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:23:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-runtime] is Ready
16:23:14 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:23:14 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:23:24	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp	Killing	Stopping container kserve-container
16:23:24	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
16:23:24	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-kserve-predictor-559bf6989 from 0 to 1
16:23:24	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-kserve-predictor-559bf6989	SuccessfulCreate	Created pod: isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv
16:23:24	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp	Killing	Stopping container kube-rbac-proxy
16:23:24	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp	Unhealthy	Readiness probe failed: Get "https://10.132.0.27:8643/healthz": dial tcp 10.132.0.27:8643: connect: connection refused
16:23:24	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp	Unhealthy	Readiness probe failed: Get "http://10.132.0.27:8080/v2/models/isvc-lightgbm-v2-runtime/ready": dial tcp 10.132.0.27:8080: connect: connection refused
16:23:24	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
16:23:25	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv	Started	Started container storage-initializer
16:23:25	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv	Created	Created container: storage-initializer
16:23:25	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:23:25	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv	AddedInterface	Add eth0 [10.132.0.28/23] from ovn-kubernetes
16:23:29	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv	Started	Started container kserve-container
16:23:29	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv	Pulled	Container image "kserve/lgbserver:latest" already present on machine
16:23:29	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv	Created	Created container: kserve-container
16:23:29	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:23:30	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv	Created	Created container: kube-rbac-proxy
16:23:30	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv	Started	Started container kube-rbac-proxy
16:23:36 (x3)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv	Unhealthy	Readiness probe failed: dial tcp 10.132.0.28:8080: connect: connection refused
16:23:39	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:23:39	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:23:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-kserve] is Ready
16:23:47 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:23:54	kserve-ci-e2e-test	deployment-controller	isvc-mlflow-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-mlflow-v2-runtime-predictor-5fdb47d546 from 0 to 1
16:23:54 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-mlflow-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-mlflow-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
16:23:54	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv	Killing	Stopping container kserve-container
16:23:54	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:23:54	kserve-ci-e2e-test	multus	isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq	AddedInterface	Add eth0 [10.132.0.29/23] from ovn-kubernetes
16:23:54	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:23:54	kserve-ci-e2e-test	replicaset-controller	isvc-mlflow-v2-runtime-predictor-5fdb47d546	SuccessfulCreate	Created pod: isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq
16:23:54	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv	Killing	Stopping container kube-rbac-proxy
16:23:54	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:23:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-mlflow-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
16:23:55	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq	Started	Started container storage-initializer
16:23:55	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq	Created	Created container: storage-initializer
16:23:59	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq	Created	Created container: kube-rbac-proxy
16:23:59	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq	Started	Started container kserve-container
16:23:59	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq	Created	Created container: kserve-container
16:23:59	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
16:23:59	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:23:59	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq	Started	Started container kube-rbac-proxy
16:24:37	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	InferenceServiceReady	InferenceService [isvc-mlflow-v2-runtime] is Ready
16:24:37 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:24:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:24:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:24:44	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-mcp-predictor-6b9b4445f6	SuccessfulCreate	Created pod: isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92
16:24:44	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq	Killing	Stopping container kserve-container
16:24:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-mcp": the object has been modified; please apply your changes to the latest version and try again
16:24:44 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-mcp": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-mcp": the object has been modified; please apply your changes to the latest version and try again
16:24:44	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-mcp-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-mcp-predictor-6b9b4445f6 from 0 to 1
16:24:44	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq	Killing	Stopping container kube-rbac-proxy
16:24:45	kserve-ci-e2e-test	multus	isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92	AddedInterface	Add eth0 [10.132.0.30/23] from ovn-kubernetes
16:24:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:24:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92	Created	Created container: storage-initializer
16:24:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92	Started	Started container storage-initializer
16:24:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1532" already present on machine
16:24:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92	Created	Created container: kserve-container
16:24:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92	Pulling	Pulling image "quay.io/opendatahub/kserve-agent:latest"
16:24:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92	Started	Started container kserve-container
16:24:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent:latest" in 2.867s (2.867s including waiting). Image size: 239444008 bytes.
16:24:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92	Created	Created container: kserve-agent
16:24:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92	Started	Started container kserve-agent
16:24:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:24:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92	Created	Created container: kube-rbac-proxy
16:24:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92	Started	Started container kube-rbac-proxy
16:26:00 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:26:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	InferenceServiceReady	InferenceService [isvc-sklearn-mcp] is Ready
16:26:04	kserve-ci-e2e-test	deployment-controller	isvc-paddle-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-predictor-6b8b7cfb4b from 0 to 1
16:26:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92	Killing	Stopping container kube-rbac-proxy
16:26:04	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-r58lh	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-paddle-predictor-serving-cert" not found
16:26:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	UpdateFailed	Failed to update status for InferenceService "isvc-paddle": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle": the object has been modified; please apply your changes to the latest version and try again
16:26:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92	Killing	Stopping container kserve-container
16:26:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92	Killing	Stopping container kserve-agent
16:26:04	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-predictor-6b8b7cfb4b	SuccessfulCreate	Created pod: isvc-paddle-predictor-6b8b7cfb4b-r58lh
16:26:05	kserve-ci-e2e-test	multus	isvc-paddle-predictor-6b8b7cfb4b-r58lh	AddedInterface	Add eth0 [10.132.0.31/23] from ovn-kubernetes
16:26:05	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-r58lh	Started	Started container storage-initializer
16:26:05	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-r58lh	Created	Created container: storage-initializer
16:26:05	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-r58lh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:26:11	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-r58lh	Pulling	Pulling image "kserve/paddleserver:latest"
16:26:23	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-r58lh	Created	Created container: kserve-container
16:26:23	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-r58lh	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:26:23	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-r58lh	Started	Started container kserve-container
16:26:23	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-r58lh	Started	Started container kube-rbac-proxy
16:26:23	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-r58lh	Created	Created container: kube-rbac-proxy
16:26:23	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-r58lh	Pulled	Successfully pulled image "kserve/paddleserver:latest" in 11.894s (11.894s including waiting). Image size: 1162550521 bytes.
16:26:30 (x4)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92	Unhealthy	Readiness probe failed: Get "http://10.132.0.30:8080/v1/models/isvc-sklearn-mcp": dial tcp 10.132.0.30:8080: connect: connection refused
16:26:30 (x6)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92	Unhealthy	Readiness probe failed: Get "https://10.132.0.30:8643/healthz": dial tcp 10.132.0.30:8643: connect: connection refused
16:26:49 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:26:49 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:26:59 (x6)	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-r58lh	Unhealthy	Readiness probe failed: dial tcp 10.132.0.31:8080: connect: connection refused
16:27:04	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:27:04	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:27:09	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	InferenceServiceReady	InferenceService [isvc-paddle] is Ready
16:27:09 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:27:16	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-r58lh	Killing	Stopping container kube-rbac-proxy
16:27:16	kserve-ci-e2e-test	deployment-controller	isvc-paddle-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-runtime-predictor-7f4d4f9dc8 from 0 to 1
16:27:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-runtime": the object has been modified; please apply your changes to the latest version and try again
16:27:16	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-runtime-predictor-7f4d4f9dc8	SuccessfulCreate	Created pod: isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm
16:27:16	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm	Created	Created container: storage-initializer
16:27:16	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:27:16	kserve-ci-e2e-test	multus	isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm	AddedInterface	Add eth0 [10.132.0.32/23] from ovn-kubernetes
16:27:16	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-r58lh	Killing	Stopping container kserve-container
16:27:16	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm	Started	Started container storage-initializer
16:27:22	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm	Started	Started container kube-rbac-proxy
16:27:22	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm	Created	Created container: kserve-container
16:27:22	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm	Started	Started container kserve-container
16:27:22	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:27:22	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm	Created	Created container: kube-rbac-proxy
16:27:22	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm	Pulled	Container image "kserve/paddleserver:latest" already present on machine
16:27:31	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:27:31	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:28:09	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	InferenceServiceReady	InferenceService [isvc-paddle-runtime] is Ready
16:28:09 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:28:16 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:28:16 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:28:17	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm	Killing	Stopping container kserve-container
16:28:17	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm	Killing	Stopping container kube-rbac-proxy
16:28:18	kserve-ci-e2e-test	multus	isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z	AddedInterface	Add eth0 [10.132.0.33/23] from ovn-kubernetes
16:28:18	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z	Created	Created container: storage-initializer
16:28:18	kserve-ci-e2e-test	deployment-controller	isvc-paddle-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-v2-kserve-predictor-7dbd59854 from 0 to 1
16:28:18	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z	Started	Started container storage-initializer
16:28:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
16:28:18	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-v2-kserve-predictor-7dbd59854	SuccessfulCreate	Created pod: isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z
16:28:18	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:28:19 (x7)	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm	Unhealthy	Readiness probe failed: dial tcp 10.132.0.32:8080: connect: connection refused
16:28:19	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm	Unhealthy	Readiness probe failed: Get "https://10.132.0.32:8643/healthz": dial tcp 10.132.0.32:8643: connect: connection refused
16:28:23	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z	Pulled	Container image "kserve/paddleserver:latest" already present on machine
16:28:23	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z	Created	Created container: kserve-container
16:28:23	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z	Started	Started container kserve-container
16:28:23	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:28:23	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z	Created	Created container: kube-rbac-proxy
16:28:23	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z	Started	Started container kube-rbac-proxy
16:28:48 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:28:48 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:29:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	InferenceServiceReady	InferenceService [isvc-paddle-v2-kserve] is Ready
16:29:11 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:29:18 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:29:18 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:29:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml": the object has been modified; please apply your changes to the latest version and try again
16:29:19 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	UpdateFailed	Failed to update status for InferenceService "isvc-pmml": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml": the object has been modified; please apply your changes to the latest version and try again
16:29:19	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-predictor-8bb578669	SuccessfulCreate	Created pod: isvc-pmml-predictor-8bb578669-4mm66
16:29:19	kserve-ci-e2e-test	deployment-controller	isvc-pmml-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-predictor-8bb578669 from 0 to 1
16:29:19	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z	Killing	Stopping container kserve-container
16:29:19	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z	Killing	Stopping container kube-rbac-proxy
16:29:20	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-4mm66	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:29:20	kserve-ci-e2e-test	multus	isvc-pmml-predictor-8bb578669-4mm66	AddedInterface	Add eth0 [10.132.0.34/23] from ovn-kubernetes
16:29:20	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-4mm66	Started	Started container storage-initializer
16:29:20	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-4mm66	Created	Created container: storage-initializer
16:29:21 (x7)	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z	Unhealthy	Readiness probe failed: dial tcp 10.132.0.33:8080: connect: connection refused
16:29:21	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z	Unhealthy	Readiness probe failed: Get "https://10.132.0.33:8643/healthz": dial tcp 10.132.0.33:8643: connect: connection refused
16:29:23	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-4mm66	Pulling	Pulling image "kserve/pmmlserver:latest"
16:29:31	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-4mm66	Created	Created container: kserve-container
16:29:31	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-4mm66	Started	Started container kserve-container
16:29:31	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-4mm66	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:29:31	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-4mm66	Created	Created container: kube-rbac-proxy
16:29:31	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-4mm66	Started	Started container kube-rbac-proxy
16:29:31	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-4mm66	Pulled	Successfully pulled image "kserve/pmmlserver:latest" in 7.303s (7.303s including waiting). Image size: 801581445 bytes.
16:29:34	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:29:34	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:30:49 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-4mm66	Unhealthy	Readiness probe failed: dial tcp 10.132.0.34:8080: connect: connection refused
16:30:49 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:30:49 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:30:58 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:30:58	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	InferenceServiceReady	InferenceService [isvc-pmml] is Ready
16:31:00	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-4mm66	Killing	Stopping container kube-rbac-proxy
16:31:00	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-4mm66	Killing	Stopping container kserve-container
16:31:01	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-lhzhb	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-pmml-runtime-predictor-serving-cert" not found
16:31:01	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-runtime-predictor-67bc544947	SuccessfulCreate	Created pod: isvc-pmml-runtime-predictor-67bc544947-lhzhb
16:31:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-runtime": the object has been modified; please apply your changes to the latest version and try again
16:31:01	kserve-ci-e2e-test	deployment-controller	isvc-pmml-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-runtime-predictor-67bc544947 from 0 to 1
16:31:02	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-lhzhb	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:31:02	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-lhzhb	Created	Created container: storage-initializer
16:31:02	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-lhzhb	Started	Started container storage-initializer
16:31:02	kserve-ci-e2e-test	multus	isvc-pmml-runtime-predictor-67bc544947-lhzhb	AddedInterface	Add eth0 [10.132.0.35/23] from ovn-kubernetes
16:31:04	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-4mm66	Unhealthy	Readiness probe failed: Get "https://10.132.0.34:8643/healthz": dial tcp 10.132.0.34:8643: connect: connection refused
16:31:06	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-lhzhb	Started	Started container kserve-container
16:31:06	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-lhzhb	Started	Started container kube-rbac-proxy
16:31:06	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-lhzhb	Created	Created container: kube-rbac-proxy
16:31:06	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-lhzhb	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:31:06	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-lhzhb	Created	Created container: kserve-container
16:31:06	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-lhzhb	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
16:31:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:31:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:32:24 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-lhzhb	Unhealthy	Readiness probe failed: dial tcp 10.132.0.35:8080: connect: connection refused
16:32:31 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:32:31 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:32:32	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	InferenceServiceReady	InferenceService [isvc-pmml-runtime] is Ready
16:32:32 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:32:42	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz	Started	Started container storage-initializer
16:32:42	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-lhzhb	Killing	Stopping container kube-rbac-proxy
16:32:42	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-lhzhb	Killing	Stopping container kserve-container
16:32:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
16:32:42	kserve-ci-e2e-test	multus	isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz	AddedInterface	Add eth0 [10.132.0.36/23] from ovn-kubernetes
16:32:42	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:32:42	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz	Created	Created container: storage-initializer
16:32:42	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-v2-kserve-predictor-6578f8ffc7	SuccessfulCreate	Created pod: isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz
16:32:42	kserve-ci-e2e-test	deployment-controller	isvc-pmml-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-v2-kserve-predictor-6578f8ffc7 from 0 to 1
16:32:42 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
16:32:44	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-lhzhb	Unhealthy	Readiness probe failed: Get "https://10.132.0.35:8643/healthz": dial tcp 10.132.0.35:8643: connect: connection refused
16:32:47	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz	Started	Started container kserve-container
16:32:47	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:32:47	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz	Created	Created container: kube-rbac-proxy
16:32:47	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz	Created	Created container: kserve-container
16:32:47	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz	Started	Started container kube-rbac-proxy
16:32:47	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
16:33:27 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:33:27 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:34:12 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:34:12 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:34:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	InferenceServiceReady	InferenceService [isvc-pmml-v2-kserve] is Ready
16:34:14 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:34:23	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-75c1b6	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-75c1b6": the object has been modified; please apply your changes to the latest version and try again
16:34:23	kserve-ci-e2e-test	replicaset-controller	isvc-primary-75c1b6-predictor-57dff89c5b	SuccessfulCreate	Created pod: isvc-primary-75c1b6-predictor-57dff89c5b-9gt85
16:34:23	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz	Killing	Stopping container kube-rbac-proxy
16:34:23	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz	Killing	Stopping container kserve-container
16:34:23	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-75c1b6	UpdateFailed	Failed to update status for InferenceService "isvc-primary-75c1b6": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-75c1b6": the object has been modified; please apply your changes to the latest version and try again
16:34:23	kserve-ci-e2e-test	deployment-controller	isvc-primary-75c1b6-predictor	ScalingReplicaSet	Scaled up replica set isvc-primary-75c1b6-predictor-57dff89c5b from 0 to 1
16:34:24 (x11)	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz	Unhealthy	Readiness probe failed: dial tcp 10.132.0.36:8080: connect: connection refused
16:34:24	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz	Unhealthy	Readiness probe failed: Get "https://10.132.0.36:8643/healthz": dial tcp 10.132.0.36:8643: connect: connection refused
16:34:24	kserve-ci-e2e-test	kubelet	isvc-primary-75c1b6-predictor-57dff89c5b-9gt85	Started	Started container storage-initializer
16:34:24	kserve-ci-e2e-test	kubelet	isvc-primary-75c1b6-predictor-57dff89c5b-9gt85	Created	Created container: storage-initializer
16:34:24	kserve-ci-e2e-test	kubelet	isvc-primary-75c1b6-predictor-57dff89c5b-9gt85	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:34:24	kserve-ci-e2e-test	multus	isvc-primary-75c1b6-predictor-57dff89c5b-9gt85	AddedInterface	Add eth0 [10.132.0.37/23] from ovn-kubernetes
16:34:27	kserve-ci-e2e-test	kubelet	isvc-primary-75c1b6-predictor-57dff89c5b-9gt85	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:34:27	kserve-ci-e2e-test	kubelet	isvc-primary-75c1b6-predictor-57dff89c5b-9gt85	Started	Started container kube-rbac-proxy
16:34:27	kserve-ci-e2e-test	kubelet	isvc-primary-75c1b6-predictor-57dff89c5b-9gt85	Created	Created container: kube-rbac-proxy
16:34:27	kserve-ci-e2e-test	kubelet	isvc-primary-75c1b6-predictor-57dff89c5b-9gt85	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1532" already present on machine
16:34:27	kserve-ci-e2e-test	kubelet	isvc-primary-75c1b6-predictor-57dff89c5b-9gt85	Started	Started container kserve-container
16:34:27	kserve-ci-e2e-test	kubelet	isvc-primary-75c1b6-predictor-57dff89c5b-9gt85	Created	Created container: kserve-container
16:35:08 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-75c1b6-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:35:08 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-75c1b6-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:35:25 (x8)	kserve-ci-e2e-test	kubelet	isvc-primary-75c1b6-predictor-57dff89c5b-9gt85	Unhealthy	Readiness probe failed: dial tcp 10.132.0.37:8080: connect: connection refused
16:35:35 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-75c1b6	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:35:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-75c1b6	InferenceServiceReady	InferenceService [isvc-primary-75c1b6] is Ready
16:35:43	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-75c1b6	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-75c1b6": the object has been modified; please apply your changes to the latest version and try again
16:35:43	kserve-ci-e2e-test	deployment-controller	isvc-secondary-75c1b6-predictor	ScalingReplicaSet	Scaled up replica set isvc-secondary-75c1b6-predictor-cdb677c59 from 0 to 1
16:35:43	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-75c1b6	UpdateFailed	Failed to update status for InferenceService "isvc-secondary-75c1b6": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-75c1b6": the object has been modified; please apply your changes to the latest version and try again
16:35:43	kserve-ci-e2e-test	kubelet	isvc-secondary-75c1b6-predictor-cdb677c59-skj8w	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-secondary-75c1b6-predictor-serving-cert" not found
16:35:43	kserve-ci-e2e-test	replicaset-controller	isvc-secondary-75c1b6-predictor-cdb677c59	SuccessfulCreate	Created pod: isvc-secondary-75c1b6-predictor-cdb677c59-skj8w
16:35:44	kserve-ci-e2e-test	multus	isvc-secondary-75c1b6-predictor-cdb677c59-skj8w	AddedInterface	Add eth0 [10.132.0.38/23] from ovn-kubernetes
16:35:49 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-75c1b6-predictor-cdb677c59-skj8w	Created	Created container: storage-initializer
16:35:49 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-75c1b6-predictor-cdb677c59-skj8w	Started	Started container storage-initializer
16:35:49 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-75c1b6-predictor-cdb677c59-skj8w	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:35:53 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-75c1b6-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:35:53 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-75c1b6-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:35:54	kserve-ci-e2e-test	kubelet	isvc-secondary-75c1b6-predictor-cdb677c59-skj8w	BackOff	Back-off restarting failed container storage-initializer in pod isvc-secondary-75c1b6-predictor-cdb677c59-skj8w_kserve-ci-e2e-test(08854e68-d614-485b-8243-b2766da71fbe)
16:35:55 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-75c1b6	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:35:58	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-75c1b6-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:35:58	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-75c1b6-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:35:59	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-919f7e	UpdateFailed	Failed to update status for InferenceService "isvc-init-fail-919f7e": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-919f7e": the object has been modified; please apply your changes to the latest version and try again
16:35:59	kserve-ci-e2e-test	kubelet	isvc-primary-75c1b6-predictor-57dff89c5b-9gt85	Killing	Stopping container kserve-container
16:35:59	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-919f7e	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-919f7e": the object has been modified; please apply your changes to the latest version and try again
16:35:59	kserve-ci-e2e-test	replicaset-controller	isvc-init-fail-919f7e-predictor-69bf6757bc	SuccessfulCreate	Created pod: isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4
16:35:59	kserve-ci-e2e-test	deployment-controller	isvc-init-fail-919f7e-predictor	ScalingReplicaSet	Scaled up replica set isvc-init-fail-919f7e-predictor-69bf6757bc from 0 to 1
16:35:59	kserve-ci-e2e-test	kubelet	isvc-primary-75c1b6-predictor-57dff89c5b-9gt85	Killing	Stopping container kube-rbac-proxy
16:36:00	kserve-ci-e2e-test	multus	isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4	AddedInterface	Add eth0 [10.132.0.39/23] from ovn-kubernetes
16:36:00	kserve-ci-e2e-test	kubelet	isvc-primary-75c1b6-predictor-57dff89c5b-9gt85	Unhealthy	Readiness probe failed: Get "https://10.132.0.37:8643/healthz": dial tcp 10.132.0.37:8643: connect: connection refused
16:36:06 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4	Started	Started container storage-initializer
16:36:06 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4	Created	Created container: storage-initializer
16:36:06 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-919f7e	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:36:06 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:36:09	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn": the object has been modified; please apply your changes to the latest version and try again
16:36:09	kserve-ci-e2e-test	kubelet	isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4	Killing	Stopping container storage-initializer
16:36:09	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-predictor-cd7c759c9	SuccessfulCreate	Created pod: isvc-predictive-sklearn-predictor-cd7c759c9-kc74j
16:36:09	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-predictor-cd7c759c9 from 0 to 1
16:36:10	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-predictor-cd7c759c9-kc74j	AddedInterface	Add eth0 [10.132.0.40/23] from ovn-kubernetes
16:36:10	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-kc74j	Started	Started container storage-initializer
16:36:10	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-kc74j	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:36:10	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-kc74j	Created	Created container: storage-initializer
16:36:15	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-kc74j	Pulling	Pulling image "kserve/predictiveserver:latest"
16:36:38	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-kc74j	Pulled	Successfully pulled image "kserve/predictiveserver:latest" in 23.24s (23.24s including waiting). Image size: 2350916147 bytes.
16:36:38	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-kc74j	Created	Created container: kserve-container
16:36:38	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-kc74j	Started	Started container kserve-container
16:36:38	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-kc74j	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:36:38	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-kc74j	Created	Created container: kube-rbac-proxy
16:36:38	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-kc74j	Started	Started container kube-rbac-proxy
16:37:09 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:37:09 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:37:48 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-kc74j	Unhealthy	Readiness probe failed: dial tcp 10.132.0.40:8080: connect: connection refused
16:37:55 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:37:55 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:37:58	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	InferenceServiceReady	InferenceService [isvc-predictive-sklearn] is Ready
16:37:58 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:38:00	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-predictor-7ff98fd74d	SuccessfulCreate	Created pod: isvc-predictive-xgboost-predictor-7ff98fd74d-868qv
16:38:00	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-kc74j	Killing	Stopping container kube-rbac-proxy
16:38:00	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-predictor-7ff98fd74d from 0 to 1
16:38:00 (x2)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-868qv	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
16:38:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost": the object has been modified; please apply your changes to the latest version and try again
16:38:00	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-kc74j	Killing	Stopping container kserve-container
16:38:01	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-kc74j	Unhealthy	Readiness probe failed: Get "https://10.132.0.40:8643/healthz": dial tcp 10.132.0.40:8643: connect: connection refused
16:38:02	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-predictor-7ff98fd74d-868qv	AddedInterface	Add eth0 [10.132.0.41/23] from ovn-kubernetes
16:38:02	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-868qv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:38:02	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-868qv	Started	Started container storage-initializer
16:38:02	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-868qv	Created	Created container: storage-initializer
16:38:06	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-868qv	Started	Started container kube-rbac-proxy
16:38:06	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-868qv	Created	Created container: kserve-container
16:38:06	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-868qv	Created	Created container: kube-rbac-proxy
16:38:06	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-868qv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:38:06	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-868qv	Started	Started container kserve-container
16:38:06	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-868qv	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
16:38:30 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:38:30 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:39:22 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:39:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	InferenceServiceReady	InferenceService [isvc-predictive-xgboost] is Ready
16:39:30	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:39:30	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-predictor-75cb94f9f from 0 to 1
16:39:30 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:39:30	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz	Created	Created container: storage-initializer
16:39:30	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz	Started	Started container storage-initializer
16:39:30	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-868qv	Killing	Stopping container kserve-container
16:39:30 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:39:30	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-868qv	Killing	Stopping container kube-rbac-proxy
16:39:30	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz	AddedInterface	Add eth0 [10.132.0.42/23] from ovn-kubernetes
16:39:30	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-predictor-75cb94f9f	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz
16:39:30	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm": the object has been modified; please apply your changes to the latest version and try again
16:39:32 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-868qv	Unhealthy	Readiness probe failed: dial tcp 10.132.0.41:8080: connect: connection refused
16:39:34	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz	Created	Created container: kserve-container
16:39:34	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz	Started	Started container kserve-container
16:39:34	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
16:39:34	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-868qv	Unhealthy	Readiness probe failed: Get "https://10.132.0.41:8643/healthz": dial tcp 10.132.0.41:8643: connect: connection refused
16:39:34	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:39:34	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz	Started	Started container kube-rbac-proxy
16:39:34	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz	Created	Created container: kube-rbac-proxy
16:40:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:40:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:40:51 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:40:51	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm] is Ready
16:41:00 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:41:00 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:41:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
16:41:00 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
16:41:00	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-v2-predictor-b5d4f6b79	SuccessfulCreate	Created pod: isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm
16:41:00	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-v2-predictor-b5d4f6b79 from 0 to 1
16:41:00	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz	Killing	Stopping container kube-rbac-proxy
16:41:00	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz	Killing	Stopping container kserve-container
16:41:01 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz	Unhealthy	Readiness probe failed: dial tcp 10.132.0.42:8080: connect: connection refused
16:41:01	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm	AddedInterface	Add eth0 [10.132.0.43/23] from ovn-kubernetes
16:41:01	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:41:01	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz	Unhealthy	Readiness probe failed: Get "https://10.132.0.42:8643/healthz": dial tcp 10.132.0.42:8643: connect: connection refused
16:41:01	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm	Created	Created container: storage-initializer
16:41:01	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm	Started	Started container storage-initializer
16:41:06	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm	Started	Started container kube-rbac-proxy
16:41:06	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
16:41:06	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm	Created	Created container: kube-rbac-proxy
16:41:06	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:41:06	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm	Started	Started container kserve-container
16:41:06	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm	Created	Created container: kserve-container
16:41:30 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:41:30 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:42:24	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	InferenceServiceReady	InferenceService [isvc-predictive-sklearn-v2] is Ready
16:42:24 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:42:30 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:42:30	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm	Killing	Stopping container kserve-container
16:42:30	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
16:42:30 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:42:30	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-v2-predictor-6577c65fd8 from 0 to 1
16:42:30	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm	Killing	Stopping container kube-rbac-proxy
16:42:31	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-v2-predictor-6577c65fd8	SuccessfulCreate	Created pod: isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n
16:42:31	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n	AddedInterface	Add eth0 [10.132.0.44/23] from ovn-kubernetes
16:42:31	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:42:31	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n	Created	Created container: storage-initializer
16:42:31	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n	Started	Started container storage-initializer
16:42:34	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm	Unhealthy	Readiness probe failed: Get "https://10.132.0.43:8643/healthz": dial tcp 10.132.0.43:8643: connect: connection refused
16:42:34 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm	Unhealthy	Readiness probe failed: Get "http://10.132.0.43:8080/v2/models/isvc-predictive-sklearn-v2/ready": dial tcp 10.132.0.43:8080: connect: connection refused
16:42:35	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
16:42:35	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n	Created	Created container: kube-rbac-proxy
16:42:35	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n	Started	Started container kube-rbac-proxy
16:42:35	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n	Created	Created container: kserve-container
16:42:35	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:42:35	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n	Started	Started container kserve-container
16:43:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:43:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:43:52	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	InferenceServiceReady	InferenceService [isvc-predictive-xgboost-v2] is Ready
16:43:52 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:44:01	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n	Killing	Stopping container kube-rbac-proxy
16:44:01	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
16:44:01	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-v2-predictor-865b4598f7	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx
16:44:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm-v2": the object has been modified; please apply your changes to the latest version and try again
16:44:01 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:44:01 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:44:01	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n	Killing	Stopping container kserve-container
16:44:01	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-v2-predictor-865b4598f7 from 0 to 1
16:44:02	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx	AddedInterface	Add eth0 [10.132.0.45/23] from ovn-kubernetes
16:44:02	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx	Started	Started container storage-initializer
16:44:02	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx	Created	Created container: storage-initializer
16:44:02	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:44:02 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n	Unhealthy	Readiness probe failed: Get "http://10.132.0.44:8080/v2/models/isvc-predictive-xgboost-v2/ready": dial tcp 10.132.0.44:8080: connect: connection refused
16:44:03	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n	Unhealthy	Readiness probe failed: Get "https://10.132.0.44:8643/healthz": dial tcp 10.132.0.44:8643: connect: connection refused
16:44:06	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:44:06	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
16:44:06	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx	Created	Created container: kserve-container
16:44:06	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx	Started	Started container kserve-container
16:44:06	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx	Started	Started container kube-rbac-proxy
16:44:06	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx	Created	Created container: kube-rbac-proxy
16:44:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:44:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:45:23	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm-v2] is Ready
16:45:23 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:45:31	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-scheduler-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-scheduler-predictor-588b969f45 from 0 to 1
16:45:31	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx	Killing	Stopping container kserve-container
16:45:31 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
16:45:31 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-scheduler": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
16:45:31	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx	Killing	Stopping container kube-rbac-proxy
16:45:31	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-scheduler-predictor-588b969f45	SuccessfulCreate	Created pod: isvc-sklearn-scheduler-predictor-588b969f45-svx42
16:45:31 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:45:31 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:45:32 (x6)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:45:33	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn": the object has been modified; please apply your changes to the latest version and try again
16:45:33	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx	Unhealthy	Readiness probe failed: Get "https://10.132.0.45:8643/healthz": dial tcp 10.132.0.45:8643: connect: connection refused
16:45:33	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-predictor-bc9cf9b57 from 0 to 1
16:45:33	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-predictor-bc9cf9b57	SuccessfulCreate	Created pod: isvc-sklearn-predictor-bc9cf9b57-k9h2x
16:45:33 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx	Unhealthy	Readiness probe failed: Get "http://10.132.0.45:8080/v2/models/isvc-predictive-lightgbm-v2/ready": dial tcp 10.132.0.45:8080: connect: connection refused
16:45:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-bc9cf9b57-k9h2x	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-predictor-serving-cert" not found
16:45:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-bc9cf9b57-k9h2x	Created	Created container: storage-initializer
16:45:34	kserve-ci-e2e-test	multus	isvc-sklearn-predictor-bc9cf9b57-k9h2x	AddedInterface	Add eth0 [10.132.0.46/23] from ovn-kubernetes
16:45:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-bc9cf9b57-k9h2x	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:45:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-bc9cf9b57-k9h2x	Started	Started container storage-initializer
16:45:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-bc9cf9b57-k9h2x	Started	Started container kserve-container
16:45:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-bc9cf9b57-k9h2x	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1532" already present on machine
16:45:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-bc9cf9b57-k9h2x	Created	Created container: kserve-container
16:45:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-bc9cf9b57-k9h2x	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:45:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-bc9cf9b57-k9h2x	Created	Created container: kube-rbac-proxy
16:45:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-bc9cf9b57-k9h2x	Started	Started container kube-rbac-proxy
16:46:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:46:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:46:45 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:46:45	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	InferenceServiceReady	InferenceService [isvc-sklearn] is Ready
16:46:48 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:46:48 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:46:53	kserve-ci-e2e-test	replicaset-controller	sklearn-v2-mlserver-predictor-65d8664766	SuccessfulCreate	Created pod: sklearn-v2-mlserver-predictor-65d8664766-llw97
16:46:53	kserve-ci-e2e-test	deployment-controller	sklearn-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set sklearn-v2-mlserver-predictor-65d8664766 from 0 to 1
16:46:53	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-llw97	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
16:46:53	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "sklearn-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "sklearn-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
16:46:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-bc9cf9b57-k9h2x	Killing	Stopping container kserve-container
16:46:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-bc9cf9b57-k9h2x	Killing	Stopping container kube-rbac-proxy
16:46:54	kserve-ci-e2e-test	multus	sklearn-v2-mlserver-predictor-65d8664766-llw97	AddedInterface	Add eth0 [10.132.0.47/23] from ovn-kubernetes
16:46:54	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-llw97	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:46:54	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-llw97	Created	Created container: storage-initializer
16:46:54	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-llw97	Started	Started container storage-initializer
16:46:55 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-bc9cf9b57-k9h2x	Unhealthy	Readiness probe failed: dial tcp 10.132.0.46:8080: connect: connection refused
16:46:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-bc9cf9b57-k9h2x	Unhealthy	Readiness probe failed: Get "https://10.132.0.46:8643/healthz": dial tcp 10.132.0.46:8643: connect: connection refused
16:46:59	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-llw97	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
16:46:59	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-llw97	Created	Created container: kube-rbac-proxy
16:46:59	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-llw97	Created	Created container: kserve-container
16:46:59	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-llw97	Started	Started container kserve-container
16:46:59	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-llw97	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:46:59	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-llw97	Started	Started container kube-rbac-proxy
16:47:08	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:47:08	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:47:36	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-llw97	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
16:47:46 (x12)	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:47:46	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	InferenceServiceReady	InferenceService [sklearn-v2-mlserver] is Ready
16:47:53	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-llw97	Killing	Stopping container kserve-container
16:47:53	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-llw97	Killing	Stopping container kube-rbac-proxy
16:47:53	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-runtime-predictor-56f6dfd9f9	SuccessfulCreate	Created pod: isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl
16:47:53 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:47:53 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:47:53	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-runtime-predictor-56f6dfd9f9 from 0 to 1
16:47:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-runtime": the object has been modified; please apply your changes to the latest version and try again
16:47:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl	Started	Started container storage-initializer
16:47:54	kserve-ci-e2e-test	multus	isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl	AddedInterface	Add eth0 [10.132.0.48/23] from ovn-kubernetes
16:47:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl	Created	Created container: storage-initializer
16:47:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:47:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
16:47:57	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-llw97	Unhealthy	Readiness probe failed: Get "http://10.132.0.47:8080/v2/models/sklearn-v2-mlserver/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
16:48:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl	Started	Started container kube-rbac-proxy
16:48:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl	Created	Created container: kserve-container
16:48:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl	Started	Started container kserve-container
16:48:01 (x2)	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-llw97	Unhealthy	Readiness probe failed: Get "https://10.132.0.47:8643/healthz": dial tcp 10.132.0.47:8643: connect: connection refused
16:48:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl	Created	Created container: kube-rbac-proxy
16:48:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:48:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1532" already present on machine
16:48:08 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl	Unhealthy	Readiness probe failed: dial tcp 10.132.0.48:8080: connect: connection refused
16:48:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-runtime] is Ready
16:48:18 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:48:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:48:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:48:30	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl	Killing	Stopping container kube-rbac-proxy
16:48:30	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl	Killing	Stopping container kserve-container
16:48:30	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-runtime-predictor-6d84c876f4	SuccessfulCreate	Created pod: isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf
16:48:30	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-runtime-predictor-6d84c876f4 from 0 to 1
16:48:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
16:48:31 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
16:48:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf	Created	Created container: storage-initializer
16:48:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:48:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf	Started	Started container storage-initializer
16:48:31	kserve-ci-e2e-test	multus	isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf	AddedInterface	Add eth0 [10.132.0.49/23] from ovn-kubernetes
16:48:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
16:48:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf	Created	Created container: kube-rbac-proxy
16:48:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf	Started	Started container kube-rbac-proxy
16:48:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:48:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf	Created	Created container: kserve-container
16:48:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf	Started	Started container kserve-container
16:48:45	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:48:45	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:49:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
16:49:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-v2-runtime] is Ready
16:49:22 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:49:30 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:49:30 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:49:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-9df9c96c7-862j2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:49:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf	Killing	Stopping container kserve-container
16:49:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf	Killing	Stopping container kube-rbac-proxy
16:49:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
16:49:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-9df9c96c7-862j2	Started	Started container storage-initializer
16:49:31	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-predictor-9df9c96c7 from 0 to 1
16:49:31	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-predictor-9df9c96c7	SuccessfulCreate	Created pod: isvc-sklearn-v2-predictor-9df9c96c7-862j2
16:49:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-9df9c96c7-862j2	Created	Created container: storage-initializer
16:49:31	kserve-ci-e2e-test	multus	isvc-sklearn-v2-predictor-9df9c96c7-862j2	AddedInterface	Add eth0 [10.132.0.50/23] from ovn-kubernetes
16:49:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf	Unhealthy	Readiness probe failed: Get "http://10.132.0.49:8080/v2/models/isvc-sklearn-v2-runtime/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
16:49:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-9df9c96c7-862j2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:49:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-9df9c96c7-862j2	Started	Started container kube-rbac-proxy
16:49:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-9df9c96c7-862j2	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1532" already present on machine
16:49:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-9df9c96c7-862j2	Created	Created container: kserve-container
16:49:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-9df9c96c7-862j2	Started	Started container kserve-container
16:49:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-9df9c96c7-862j2	Created	Created container: kube-rbac-proxy
16:49:37 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf	Unhealthy	Readiness probe failed: Get "https://10.132.0.49:8643/healthz": dial tcp 10.132.0.49:8643: connect: connection refused
16:49:46	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:49:46	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:50:42 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:50:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	InferenceServiceReady	InferenceService [isvc-sklearn-v2] is Ready
16:50:46 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:50:46 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:50:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:50:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72	Started	Started container storage-initializer
16:50:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72	Created	Created container: storage-initializer
16:50:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-9df9c96c7-862j2	Killing	Stopping container kube-rbac-proxy
16:50:51	kserve-ci-e2e-test	multus	isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72	AddedInterface	Add eth0 [10.132.0.51/23] from ovn-kubernetes
16:50:51	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-mixed": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-mixed": the object has been modified; please apply your changes to the latest version and try again
16:50:51	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-mixed-predictor-5cd9489bbd	SuccessfulCreate	Created pod: isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72
16:50:51	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-mixed-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-mixed-predictor-5cd9489bbd from 0 to 1
16:50:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-9df9c96c7-862j2	Killing	Stopping container kserve-container
16:50:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-9df9c96c7-862j2	Unhealthy	Readiness probe failed: Get "https://10.132.0.50:8643/healthz": dial tcp 10.132.0.50:8643: connect: connection refused
16:50:52 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-9df9c96c7-862j2	Unhealthy	Readiness probe failed: dial tcp 10.132.0.50:8080: connect: connection refused
16:50:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72	Started	Started container kube-rbac-proxy
16:50:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72	Created	Created container: kube-rbac-proxy
16:50:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1532" already present on machine
16:50:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72	Created	Created container: kserve-container
16:50:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72	Started	Started container kserve-container
16:50:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:51:36 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:51:36 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:52:03 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:52:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	InferenceServiceReady	InferenceService [isvc-sklearn-v2-mixed] is Ready
16:52:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:52:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:52:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow": the object has been modified; please apply your changes to the latest version and try again
16:52:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72	Killing	Stopping container kserve-container
16:52:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72	Killing	Stopping container kube-rbac-proxy
16:52:11	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-predictor-6756f669d7 from 0 to 1
16:52:11	kserve-ci-e2e-test	multus	isvc-tensorflow-predictor-6756f669d7-78wr2	AddedInterface	Add eth0 [10.132.0.52/23] from ovn-kubernetes
16:52:11	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-78wr2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:52:11	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-predictor-6756f669d7	SuccessfulCreate	Created pod: isvc-tensorflow-predictor-6756f669d7-78wr2
16:52:12	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-78wr2	Created	Created container: storage-initializer
16:52:12	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-78wr2	Started	Started container storage-initializer
16:52:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72	Unhealthy	Readiness probe failed: Get "https://10.132.0.51:8643/healthz": dial tcp 10.132.0.51:8643: connect: connection refused
16:52:13 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72	Unhealthy	Readiness probe failed: dial tcp 10.132.0.51:8080: connect: connection refused
16:52:16	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-78wr2	Pulling	Pulling image "tensorflow/serving:2.6.2"
16:52:20	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-78wr2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:52:20	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-78wr2	Started	Started container kserve-container
16:52:20	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-78wr2	Created	Created container: kube-rbac-proxy
16:52:20	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-78wr2	Started	Started container kube-rbac-proxy
16:52:20	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-78wr2	Created	Created container: kserve-container
16:52:20	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-78wr2	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 3.628s (3.628s including waiting). Image size: 425873876 bytes.
16:52:27 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-78wr2	Unhealthy	Readiness probe failed: dial tcp 10.132.0.52:8080: connect: connection refused
16:52:38 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:52:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	InferenceServiceReady	InferenceService [isvc-tensorflow] is Ready
16:52:41 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:52:41 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:52:52	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-runtime-predictor-8699d78cf	SuccessfulCreate	Created pod: isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl
16:52:52	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-runtime-predictor-8699d78cf from 0 to 1
16:52:52	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow-runtime": the object has been modified; please apply your changes to the latest version and try again
16:52:52	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-78wr2	Killing	Stopping container kube-rbac-proxy
16:52:52	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-78wr2	Killing	Stopping container kserve-container
16:52:53	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl	Started	Started container storage-initializer
16:52:53	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:52:53	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl	Created	Created container: storage-initializer
16:52:53	kserve-ci-e2e-test	multus	isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl	AddedInterface	Add eth0 [10.132.0.53/23] from ovn-kubernetes
16:52:53	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
16:52:59	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:52:59	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl	Pulled	Container image "tensorflow/serving:2.6.2" already present on machine
16:52:59	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl	Started	Started container kube-rbac-proxy
16:52:59	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl	Created	Created container: kube-rbac-proxy
16:52:59	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl	Started	Started container kserve-container
16:52:59	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl	Created	Created container: kserve-container
16:53:07	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:53:07 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl	Unhealthy	Readiness probe failed: dial tcp 10.132.0.53:8080: connect: connection refused
16:53:07	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:53:17 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:53:17	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	InferenceServiceReady	InferenceService [isvc-tensorflow-runtime] is Ready
16:53:22	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:53:22	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:53:22 (x7)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-78wr2	Unhealthy	Readiness probe failed: Get "https://10.132.0.52:8643/healthz": dial tcp 10.132.0.52:8643: connect: connection refused
16:53:34	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl	Killing	Stopping container kserve-container
16:53:34	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl	Killing	Stopping container kube-rbac-proxy
16:53:34	kserve-ci-e2e-test	replicaset-controller	isvc-triton-predictor-84bb65d94b	SuccessfulCreate	Created pod: isvc-triton-predictor-84bb65d94b-wg8gk
16:53:34	kserve-ci-e2e-test	deployment-controller	isvc-triton-predictor	ScalingReplicaSet	Scaled up replica set isvc-triton-predictor-84bb65d94b from 0 to 1
16:53:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	UpdateFailed	Failed to update status for InferenceService "isvc-triton": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-triton": the object has been modified; please apply your changes to the latest version and try again
16:53:34	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-wg8gk	Started	Started container storage-initializer
16:53:34	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-wg8gk	Created	Created container: storage-initializer
16:53:34	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-wg8gk	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:53:34	kserve-ci-e2e-test	multus	isvc-triton-predictor-84bb65d94b-wg8gk	AddedInterface	Add eth0 [10.132.0.54/23] from ovn-kubernetes
16:53:38	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-wg8gk	Pulling	Pulling image "nvcr.io/nvidia/tritonserver:23.05-py3"
16:54:02 (x6)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl	Unhealthy	Readiness probe failed: Get "https://10.132.0.53:8643/healthz": dial tcp 10.132.0.53:8643: connect: connection refused
16:55:32	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-wg8gk	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:55:32	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-wg8gk	Created	Created container: kube-rbac-proxy
16:55:32	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-wg8gk	Started	Started container kube-rbac-proxy
16:55:32	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-wg8gk	Pulled	Successfully pulled image "nvcr.io/nvidia/tritonserver:23.05-py3" in 1m54.53s (1m54.53s including waiting). Image size: 12907074623 bytes.
16:55:32	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-wg8gk	Started	Started container kserve-container
16:55:32	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-wg8gk	Created	Created container: kserve-container
16:55:34 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:55:34 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:55:35 (x2)	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-wg8gk	Unhealthy	Readiness probe failed: dial tcp 10.132.0.54:8080: connect: connection refused
16:55:40 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:55:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	InferenceServiceReady	InferenceService [isvc-triton] is Ready
16:55:45	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-wg8gk	Killing	Stopping container kube-rbac-proxy
16:55:45	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost": the object has been modified; please apply your changes to the latest version and try again
16:55:45	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-wg8gk	Killing	Stopping container kserve-container
16:55:45	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-predictor-8689c4cfcc from 0 to 1
16:55:45	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-predictor-8689c4cfcc	SuccessfulCreate	Created pod: isvc-xgboost-predictor-8689c4cfcc-rtqwj
16:55:46	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-rtqwj	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-predictor-serving-cert" not found
16:55:47	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-rtqwj	Created	Created container: storage-initializer
16:55:47	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-rtqwj	Started	Started container storage-initializer
16:55:47	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-rtqwj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:55:47	kserve-ci-e2e-test	multus	isvc-xgboost-predictor-8689c4cfcc-rtqwj	AddedInterface	Add eth0 [10.132.0.55/23] from ovn-kubernetes
16:55:51	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-rtqwj	Pulling	Pulling image "kserve/xgbserver:latest"
16:56:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-rtqwj	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 19.405s (19.405s including waiting). Image size: 1306238203 bytes.
16:56:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-rtqwj	Created	Created container: kserve-container
16:56:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-rtqwj	Started	Started container kserve-container
16:56:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-rtqwj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:56:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-rtqwj	Created	Created container: kube-rbac-proxy
16:56:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-rtqwj	Started	Started container kube-rbac-proxy
16:56:45 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:56:45 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:57:15 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:57:15 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:57:18 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:57:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	InferenceServiceReady	InferenceService [isvc-xgboost] is Ready
16:57:25	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-rtqwj	Killing	Stopping container kube-rbac-proxy
16:57:25	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-rtqwj	Killing	Stopping container kserve-container
16:57:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd	Created	Created container: storage-initializer
16:57:26	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
16:57:26	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-mlserver-predictor-67d4bc6646	SuccessfulCreate	Created pod: isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd
16:57:26	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-mlserver-predictor-67d4bc6646 from 0 to 1
16:57:26 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
16:57:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:57:26	kserve-ci-e2e-test	multus	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd	AddedInterface	Add eth0 [10.132.0.56/23] from ovn-kubernetes
16:57:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd	Started	Started container storage-initializer
16:57:28 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-rtqwj	Unhealthy	Readiness probe failed: dial tcp 10.132.0.55:8080: connect: connection refused
16:57:28	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-rtqwj	Unhealthy	Readiness probe failed: Get "https://10.132.0.55:8643/healthz": dial tcp 10.132.0.55:8643: connect: connection refused
16:57:30	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd	Started	Started container kube-rbac-proxy
16:57:30	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
16:57:30	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd	Created	Created container: kserve-container
16:57:30	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd	Created	Created container: kube-rbac-proxy
16:57:30	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:57:30	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd	Started	Started container kserve-container
16:57:56	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:57:56	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:58:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	InferenceServiceReady	InferenceService [isvc-xgboost-v2-mlserver] is Ready
16:58:09 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:58:11 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:58:11 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:58:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd	Killing	Stopping container kube-rbac-proxy
16:58:16	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-ppp6x	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "xgboost-v2-mlserver-predictor-serving-cert" not found
16:58:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd	Killing	Stopping container kserve-container
16:58:16	kserve-ci-e2e-test	replicaset-controller	xgboost-v2-mlserver-predictor-7799869d6f	SuccessfulCreate	Created pod: xgboost-v2-mlserver-predictor-7799869d6f-ppp6x
16:58:16	kserve-ci-e2e-test	deployment-controller	xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set xgboost-v2-mlserver-predictor-7799869d6f from 0 to 1
16:58:16	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
16:58:17	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-ppp6x	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:58:17	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-ppp6x	Created	Created container: storage-initializer
16:58:17	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-ppp6x	Started	Started container storage-initializer
16:58:17	kserve-ci-e2e-test	multus	xgboost-v2-mlserver-predictor-7799869d6f-ppp6x	AddedInterface	Add eth0 [10.132.0.57/23] from ovn-kubernetes
16:58:18	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd	Unhealthy	Readiness probe failed: Get "http://10.132.0.56:8080/v2/models/isvc-xgboost-v2-mlserver/ready": dial tcp 10.132.0.56:8080: connect: connection refused
16:58:18	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd	Unhealthy	Readiness probe failed: Get "https://10.132.0.56:8643/healthz": dial tcp 10.132.0.56:8643: connect: connection refused
16:58:22	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-ppp6x	Started	Started container kserve-container
16:58:22	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-ppp6x	Created	Created container: kserve-container
16:58:22	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-ppp6x	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
16:58:22	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-ppp6x	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:58:22	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-ppp6x	Started	Started container kube-rbac-proxy
16:58:22	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-ppp6x	Created	Created container: kube-rbac-proxy
16:58:46 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:58:46 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:58:59 (x12)	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:58:59	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	InferenceServiceReady	InferenceService [xgboost-v2-mlserver] is Ready
16:59:01	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:59:01	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:59:06	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-fmnv4	Created	Created container: storage-initializer
16:59:06	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-ppp6x	Killing	Stopping container kserve-container
16:59:06	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-ppp6x	Killing	Stopping container kube-rbac-proxy
16:59:06	kserve-ci-e2e-test	multus	isvc-xgboost-runtime-predictor-779db84d9-fmnv4	AddedInterface	Add eth0 [10.132.0.58/23] from ovn-kubernetes
16:59:06	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-fmnv4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
16:59:06	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-fmnv4	Started	Started container storage-initializer
16:59:06	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-runtime-predictor-779db84d9	SuccessfulCreate	Created pod: isvc-xgboost-runtime-predictor-779db84d9-fmnv4
16:59:06	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-runtime-predictor-779db84d9 from 0 to 1
16:59:06	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-runtime": the object has been modified; please apply your changes to the latest version and try again
16:59:09	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-ppp6x	Unhealthy	Readiness probe failed: Get "https://10.132.0.57:8643/healthz": dial tcp 10.132.0.57:8643: connect: connection refused
16:59:09	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-ppp6x	Unhealthy	Readiness probe failed: Get "http://10.132.0.57:8080/v2/models/xgboost-v2-mlserver/ready": dial tcp 10.132.0.57:8080: connect: connection refused
16:59:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-fmnv4	Started	Started container kube-rbac-proxy
16:59:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-fmnv4	Pulled	Container image "kserve/xgbserver:latest" already present on machine
16:59:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-fmnv4	Created	Created container: kube-rbac-proxy
16:59:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-fmnv4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
16:59:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-fmnv4	Started	Started container kserve-container
16:59:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-fmnv4	Created	Created container: kserve-container
16:59:51 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:59:51 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:00:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-runtime] is Ready
17:00:19 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:00:21 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:00:21 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:00:26	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-runtime-predictor-6dc5954dc	SuccessfulCreate	Created pod: isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4
17:00:26	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-runtime-predictor-6dc5954dc from 0 to 1
17:00:26	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
17:00:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
17:00:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-fmnv4	Killing	Stopping container kserve-container
17:00:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-fmnv4	Killing	Stopping container kube-rbac-proxy
17:00:27	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4	Started	Started container storage-initializer
17:00:27	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4	Created	Created container: storage-initializer
17:00:27	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
17:00:27	kserve-ci-e2e-test	multus	isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4	AddedInterface	Add eth0 [10.132.0.59/23] from ovn-kubernetes
17:00:29 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-fmnv4	Unhealthy	Readiness probe failed: dial tcp 10.132.0.58:8080: connect: connection refused
17:00:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-fmnv4	Unhealthy	Readiness probe failed: Get "https://10.132.0.58:8643/healthz": dial tcp 10.132.0.58:8643: connect: connection refused
17:00:32	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4	Created	Created container: kserve-container
17:00:32	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4	Started	Started container kserve-container
17:00:32	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4	Started	Started container kube-rbac-proxy
17:00:32	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:00:32	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
17:00:32	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4	Created	Created container: kube-rbac-proxy
17:01:10	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
17:01:11 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:01:11 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:01:20 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:01:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-v2-runtime] is Ready
17:01:26 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:01:26	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
17:01:26 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
17:01:26	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-predictor-6fcdd6977c	SuccessfulCreate	Created pod: isvc-xgboost-v2-predictor-6fcdd6977c-78tfs
17:01:26	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-predictor-6fcdd6977c from 0 to 1
17:01:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4	Killing	Stopping container kserve-container
17:01:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4	Killing	Stopping container kube-rbac-proxy
17:01:26 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:01:27	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-78tfs	Started	Started container storage-initializer
17:01:27	kserve-ci-e2e-test	multus	isvc-xgboost-v2-predictor-6fcdd6977c-78tfs	AddedInterface	Add eth0 [10.132.0.60/23] from ovn-kubernetes
17:01:27	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-78tfs	Created	Created container: storage-initializer
17:01:27	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-78tfs	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
17:01:30	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4	Unhealthy	Readiness probe failed: Get "http://10.132.0.59:8080/v2/models/isvc-xgboost-v2-runtime/ready": dial tcp 10.132.0.59:8080: connect: connection refused
17:01:30	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4	Unhealthy	Readiness probe failed: Get "https://10.132.0.59:8643/healthz": dial tcp 10.132.0.59:8643: connect: connection refused
17:01:31	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-78tfs	Pulled	Container image "kserve/xgbserver:latest" already present on machine
17:01:31	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-78tfs	Started	Started container kube-rbac-proxy
17:01:31	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-78tfs	Created	Created container: kube-rbac-proxy
17:01:31	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-78tfs	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:01:31	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-78tfs	Started	Started container kserve-container
17:01:31	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-78tfs	Created	Created container: kserve-container
17:02:26 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:02:26 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:02:39 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:02:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	InferenceServiceReady	InferenceService [isvc-xgboost-v2] is Ready
17:02:41	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:02:41	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:02:46	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-78tfs	Killing	Stopping container kube-rbac-proxy
17:02:46	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-predictor-89bbdf6fb	SuccessfulCreate	Created pod: isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5
17:02:46	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-predictor-89bbdf6fb from 0 to 1
17:02:46	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3": the object has been modified; please apply your changes to the latest version and try again
17:02:46	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-78tfs	Killing	Stopping container kserve-container
17:02:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5	Started	Started container storage-initializer
17:02:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5	Created	Created container: storage-initializer
17:02:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
17:02:47	kserve-ci-e2e-test	multus	isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5	AddedInterface	Add eth0 [10.132.0.61/23] from ovn-kubernetes
17:02:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5	Created	Created container: kube-rbac-proxy
17:02:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5	Started	Started container kserve-container
17:02:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1532" already present on machine
17:02:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5	Created	Created container: kserve-container
17:02:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:02:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5	Started	Started container kube-rbac-proxy
17:02:49 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-78tfs	Unhealthy	Readiness probe failed: dial tcp 10.132.0.60:8080: connect: connection refused
17:02:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-78tfs	Unhealthy	Readiness probe failed: Get "https://10.132.0.60:8643/healthz": dial tcp 10.132.0.60:8643: connect: connection refused
17:03:16 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:03:16 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:03:46 (x8)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5	Unhealthy	Readiness probe failed: dial tcp 10.132.0.61:8080: connect: connection refused
17:03:56 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:03:56	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	InferenceServiceReady	InferenceService [isvc-sklearn-s3] is Ready
17:04:01 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:04:01 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:04:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5	Killing	Stopping container kube-rbac-proxy
17:04:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5	Killing	Stopping container kserve-container
17:04:07	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm	AddedInterface	Add eth0 [10.132.0.62/23] from ovn-kubernetes
17:04:07	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm
17:04:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm	Created	Created container: storage-initializer
17:04:07	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-pass": the object has been modified; please apply your changes to the latest version and try again
17:04:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm	Started	Started container storage-initializer
17:04:07 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-pass": the object has been modified; please apply your changes to the latest version and try again
17:04:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
17:04:07	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b from 0 to 1
17:04:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm	Created	Created container: kserve-container
17:04:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm	Started	Started container kube-rbac-proxy
17:04:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm	Created	Created container: kube-rbac-proxy
17:04:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:04:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm	Started	Started container kserve-container
17:04:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1532" already present on machine
17:04:22	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:04:22	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:05:07 (x8)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm	Unhealthy	Readiness probe failed: dial tcp 10.132.0.62:8080: connect: connection refused
17:05:17 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:05:17	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-global-pass] is Ready
17:05:22 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:05:22 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:05:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm	Killing	Stopping container kube-rbac-proxy
17:05:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm	Killing	Stopping container kserve-container
17:05:28	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4 from 0 to 1
17:05:28	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db
17:05:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-fail": the object has been modified; please apply your changes to the latest version and try again
17:05:28	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db	AddedInterface	Add eth0 [10.132.0.63/23] from ovn-kubernetes
17:05:34 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db	Created	Created container: storage-initializer
17:05:34 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
17:05:34 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db	Started	Started container storage-initializer
17:05:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db	BackOff	Back-off restarting failed container storage-initializer in pod isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db_kserve-ci-e2e-test(850d0ee0-b3bf-4a8f-853b-a88807a481a7)
17:05:37 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:05:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-pass": the object has been modified; please apply your changes to the latest version and try again
17:05:39	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb from 0 to 1
17:05:39	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq
17:05:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq	Started	Started container storage-initializer
17:05:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq	Created	Created container: storage-initializer
17:05:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
17:05:39	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq	AddedInterface	Add eth0 [10.132.0.64/23] from ovn-kubernetes
17:05:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq	Started	Started container kserve-container
17:05:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1532" already present on machine
17:05:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq	Created	Created container: kserve-container
17:05:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:05:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq	Created	Created container: kube-rbac-proxy
17:05:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq	Started	Started container kube-rbac-proxy
17:06:39 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:06:39 (x8)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq	Unhealthy	Readiness probe failed: dial tcp 10.132.0.64:8080: connect: connection refused
17:06:39 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:06:49 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:06:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-custom-pass] is Ready
17:06:54	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:06:54	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:06:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq	Unhealthy	Readiness probe failed: Get "https://10.132.0.64:8643/healthz": dial tcp 10.132.0.64:8643: connect: connection refused
17:06:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq	Killing	Stopping container kube-rbac-proxy
17:06:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq	Killing	Stopping container kserve-container
17:07:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-fail": the object has been modified; please apply your changes to the latest version and try again
17:07:00	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284
17:07:00	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284	AddedInterface	Add eth0 [10.132.0.65/23] from ovn-kubernetes
17:07:00	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d from 0 to 1
17:07:05 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284	Started	Started container storage-initializer
17:07:05 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284	Created	Created container: storage-initializer
17:07:05 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
17:07:05 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:07:10	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284	Killing	Stopping container storage-initializer
17:07:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-pass": the object has been modified; please apply your changes to the latest version and try again
17:07:11	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-pass-predictor-754974974c from 0 to 1
17:07:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
17:07:11	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-pass-predictor-754974974c	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v
17:07:12	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v	AddedInterface	Add eth0 [10.132.0.66/23] from ovn-kubernetes
17:07:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
17:07:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v	Created	Created container: storage-initializer
17:07:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v	Started	Started container storage-initializer
17:07:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v	Created	Created container: kube-rbac-proxy
17:07:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1532" already present on machine
17:07:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v	Started	Started container kube-rbac-proxy
17:07:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:07:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v	Started	Started container kserve-container
17:07:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v	Created	Created container: kserve-container
17:07:26	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:07:26	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:08:21	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-serving-pass] is Ready
17:08:21 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:08:26 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:08:26 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:08:31 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v	Unhealthy	Readiness probe failed: dial tcp 10.132.0.66:8080: connect: connection refused
17:08:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v	Killing	Stopping container kserve-container
17:08:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v	Killing	Stopping container kube-rbac-proxy
17:08:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v	Unhealthy	Readiness probe failed: Get "https://10.132.0.66:8643/healthz": dial tcp 10.132.0.66:8643: connect: connection refused
17:08:32	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-fail": the object has been modified; please apply your changes to the latest version and try again
17:08:32	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8	AddedInterface	Add eth0 [10.132.0.67/23] from ovn-kubernetes
17:08:32	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9 from 0 to 1
17:08:32	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8
17:08:37 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8	Created	Created container: storage-initializer
17:08:37 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8	Started	Started container storage-initializer
17:08:37 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0413900e48a4ba69f75c45b06af88f792921f4669631dcf3a047c1fe5c1fc341" already present on machine
17:08:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8	BackOff	Back-off restarting failed container storage-initializer in pod isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8_kserve-ci-e2e-test(060cbf2e-497d-405a-8a05-c129289d9385)
17:08:41 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.