Events

Time	Namespace	Component	RelatedObject	Reason	Message
15:17:39	kserve-ci-e2e-test		isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-bcss7	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-bcss7 to ip-10-0-129-76.ec2.internal
15:50:44	kserve-ci-e2e-test		xgboost-v2-mlserver-predictor-c58d48f-wjp8w	Scheduled	Successfully assigned kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-c58d48f-wjp8w to ip-10-0-129-76.ec2.internal
15:40:00	kserve-ci-e2e-test		sklearn-v2-mlserver-predictor-695d5f5568-krhvs	Scheduled	Successfully assigned kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-krhvs to ip-10-0-129-76.ec2.internal
15:23:14	kserve-ci-e2e-test		isvc-pmml-predictor-5584ffd8c9-ft682	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-ft682 to ip-10-0-139-55.ec2.internal
15:24:55	kserve-ci-e2e-test		isvc-pmml-runtime-predictor-7576f6b69f-zt4tv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-zt4tv to ip-10-0-139-55.ec2.internal
15:29:36	kserve-ci-e2e-test		isvc-init-fail-db90c5-predictor-cd8bb7b7d-ldt9w	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-init-fail-db90c5-predictor-cd8bb7b7d-ldt9w to ip-10-0-129-76.ec2.internal
15:21:11	kserve-ci-e2e-test		isvc-paddle-runtime-predictor-86b49c4466-xklbk	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-xklbk to ip-10-0-139-55.ec2.internal
15:26:26	kserve-ci-e2e-test		isvc-pmml-v2-kserve-predictor-75b87ff64c-24hvt	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-24hvt to ip-10-0-139-55.ec2.internal
15:09:52	kserve-ci-e2e-test		message-dumper-predictor-7f66cccfb6-2ttq5	Scheduled	Successfully assigned kserve-ci-e2e-test/message-dumper-predictor-7f66cccfb6-2ttq5 to ip-10-0-139-101.ec2.internal
15:52:34	kserve-ci-e2e-test		isvc-xgboost-v2-runtime-predictor-b5d6966c7-g9rnd	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-g9rnd to ip-10-0-129-76.ec2.internal
15:53:34	kserve-ci-e2e-test		isvc-xgboost-v2-predictor-5db5686f9f-thlnv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-thlnv to ip-10-0-139-55.ec2.internal
15:11:28	kserve-ci-e2e-test		isvc-lightgbm-predictor-78c8d484d6-xghcj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-xghcj to ip-10-0-129-76.ec2.internal
15:50:03	kserve-ci-e2e-test		isvc-xgboost-v2-mlserver-predictor-58b7db6668-khmcs	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-58b7db6668-khmcs to ip-10-0-129-76.ec2.internal
15:51:24	kserve-ci-e2e-test		isvc-xgboost-runtime-predictor-687c7765c9-sqv4p	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-sqv4p to ip-10-0-139-55.ec2.internal
15:48:33	kserve-ci-e2e-test		isvc-xgboost-predictor-6dbc9d6d47-rnkmt	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-rnkmt to ip-10-0-139-55.ec2.internal
15:46:11	kserve-ci-e2e-test		isvc-triton-predictor-85f9f46646-4rt76	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-4rt76 to ip-10-0-139-55.ec2.internal
15:45:40	kserve-ci-e2e-test		isvc-tensorflow-runtime-predictor-854bdff69c-s2rdr	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-s2rdr to ip-10-0-139-55.ec2.internal
15:44:58	kserve-ci-e2e-test		isvc-tensorflow-predictor-88f6f6cb7-fjkvt	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-fjkvt to ip-10-0-139-55.ec2.internal
15:41:28	kserve-ci-e2e-test		isvc-sklearn-v2-runtime-predictor-7dd9b85c64-c8wt2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-c8wt2 to ip-10-0-129-76.ec2.internal
15:19:59	kserve-ci-e2e-test		isvc-paddle-predictor-7dddcb4bd4-h6xds	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-h6xds to ip-10-0-129-76.ec2.internal
15:27:58	kserve-ci-e2e-test		isvc-primary-21dbf3-predictor-6bd6957cff-swgrv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-primary-21dbf3-predictor-6bd6957cff-swgrv to ip-10-0-129-76.ec2.internal
15:29:18	kserve-ci-e2e-test		isvc-secondary-21dbf3-predictor-78d7c59b5d-rfdhs	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-secondary-21dbf3-predictor-78d7c59b5d-rfdhs to ip-10-0-129-76.ec2.internal
15:42:18	kserve-ci-e2e-test		isvc-sklearn-v2-predictor-84556fcc44-2smfh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84556fcc44-2smfh to ip-10-0-129-76.ec2.internal
15:43:38	kserve-ci-e2e-test		isvc-sklearn-v2-mixed-predictor-56d7d747c9-zlrct	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56d7d747c9-zlrct to ip-10-0-129-76.ec2.internal
15:59:09	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-pass-predictor-64ddbd787d-g8vmr	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-64ddbd787d-g8vmr to ip-10-0-129-76.ec2.internal
16:00:20	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-fail-predictor-5b99d6b556-zqc82	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5b99d6b556-zqc82 to ip-10-0-129-76.ec2.internal
15:32:56	kserve-ci-e2e-test		isvc-predictive-lightgbm-predictor-669896799c-zvdk7	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-zvdk7 to ip-10-0-139-55.ec2.internal
15:56:05	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-pass-predictor-7dcd76cc75-hsrgp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7dcd76cc75-hsrgp to ip-10-0-129-76.ec2.internal
15:13:08	kserve-ci-e2e-test		isvc-lightgbm-runtime-predictor-64984c7cb-dwxrv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-dwxrv to ip-10-0-129-76.ec2.internal
15:08:27	kserve-ci-e2e-test		isvc-sklearn-batcher-custom-predictor-564b94bff5-tjgs2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-564b94bff5-tjgs2 to ip-10-0-129-76.ec2.internal
15:57:26	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-fail-predictor-76467cdc57-xcm9j	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-76467cdc57-xcm9j to ip-10-0-129-76.ec2.internal
15:57:37	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-pass-predictor-65cf754f99-lp4l7	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65cf754f99-lp4l7 to ip-10-0-129-76.ec2.internal
15:58:58	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-fail-predictor-746ccb9c65-kspkv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-746ccb9c65-kspkv to ip-10-0-129-76.ec2.internal
15:54:45	kserve-ci-e2e-test		isvc-sklearn-s3-predictor-54d746bb87-gk6gt	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54d746bb87-gk6gt to ip-10-0-129-76.ec2.internal
15:40:50	kserve-ci-e2e-test		isvc-sklearn-runtime-predictor-9c688ddc4-nlk27	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9c688ddc4-nlk27 to ip-10-0-129-76.ec2.internal
15:38:40	kserve-ci-e2e-test		isvc-sklearn-predictor-74df74d548-5559r	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-predictor-74df74d548-5559r to ip-10-0-129-76.ec2.internal
15:37:08	kserve-ci-e2e-test		isvc-predictive-lightgbm-v2-predictor-65df45cf79-sbblm	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-sbblm to ip-10-0-139-55.ec2.internal
15:18:39	kserve-ci-e2e-test		isvc-sklearn-mcp-predictor-65975766bf-cz4hx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-65975766bf-cz4hx to ip-10-0-129-76.ec2.internal
15:31:36	kserve-ci-e2e-test		isvc-predictive-xgboost-predictor-794b54b9b4-hxq4v	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-hxq4v to ip-10-0-139-55.ec2.internal
15:22:22	kserve-ci-e2e-test		isvc-paddle-v2-kserve-predictor-7c6bd76f7b-bc22l	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-bc22l to ip-10-0-139-55.ec2.internal
15:17:59	kserve-ci-e2e-test		isvc-mlflow-v2-runtime-predictor-5497fc5f68-lxbwj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5497fc5f68-lxbwj to ip-10-0-129-76.ec2.internal
15:10:03	kserve-ci-e2e-test		isvc-logger-predictor-7c7767ff77-fcpnf	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-logger-predictor-7c7767ff77-fcpnf to ip-10-0-129-76.ec2.internal
15:35:47	kserve-ci-e2e-test		isvc-predictive-xgboost-v2-predictor-67fd65d6cb-vpbbk	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-vpbbk to ip-10-0-139-55.ec2.internal
15:29:46	kserve-ci-e2e-test		isvc-predictive-sklearn-predictor-85bccb8945-hkrrp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-hkrrp to ip-10-0-139-55.ec2.internal
15:14:39	kserve-ci-e2e-test		isvc-lightgbm-v2-runtime-predictor-8d44c64dc-7kmd5	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8d44c64dc-7kmd5 to ip-10-0-129-76.ec2.internal
15:34:27	kserve-ci-e2e-test		isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-68rw9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-68rw9 to ip-10-0-139-55.ec2.internal
15:06:42	kserve-ci-e2e-test		isvc-sklearn-batcher-predictor-5d8576b666-c782k	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8576b666-c782k to ip-10-0-129-76.ec2.internal
15:06:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
15:06:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
15:06:42	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-predictor-5d8576b666 from 0 to 1
15:06:42	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-predictor-5d8576b666	SuccessfulCreate	Created pod: isvc-sklearn-batcher-predictor-5d8576b666-c782k
15:06:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-5d8576b666-c782k	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a"
15:06:43	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-predictor-5d8576b666-c782k	AddedInterface	Add eth0 [10.134.0.33/23] from ovn-kubernetes
15:06:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-5d8576b666-c782k	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" in 3.288s (3.288s including waiting). Image size: 300960724 bytes.
15:06:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-5d8576b666-c782k	Created	Created container: storage-initializer
15:06:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-5d8576b666-c782k	Started	Started container storage-initializer
15:06:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-5d8576b666-c782k	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1407"
15:07:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-5d8576b666-c782k	Pulling	Pulling image "quay.io/opendatahub/kserve-agent@sha256:93a3c88e5cf08b3e8030b29f6d098cd5406a6fd3ad769e6e697c2ae6303a7cab"
15:07:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-5d8576b666-c782k	Created	Created container: kserve-container
15:07:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-5d8576b666-c782k	Started	Started container kserve-container
15:07:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-5d8576b666-c782k	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1407" in 13.168s (13.168s including waiting). Image size: 1560613827 bytes.
15:07:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-5d8576b666-c782k	Started	Started container agent
15:07:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-5d8576b666-c782k	Created	Created container: agent
15:07:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-5d8576b666-c782k	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:93a3c88e5cf08b3e8030b29f6d098cd5406a6fd3ad769e6e697c2ae6303a7cab" in 3.125s (3.125s including waiting). Image size: 237895798 bytes.
15:07:42 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:07:42 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:08:12 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:08:12 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:08:20 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:08:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InferenceServiceReady	InferenceService [isvc-sklearn-batcher] is Ready
15:08:27	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetScale	deployments.apps "isvc-sklearn-batcher-predictor" not found
15:08:27	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher-custom": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher-custom": the object has been modified; please apply your changes to the latest version and try again
15:08:27	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-custom-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-custom-predictor-564b94bff5 from 0 to 1
15:08:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-5d8576b666-c782k	Killing	Stopping container kserve-container
15:08:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-5d8576b666-c782k	Killing	Stopping container agent
15:08:27	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-custom-predictor-564b94bff5	SuccessfulCreate	Created pod: isvc-sklearn-batcher-custom-predictor-564b94bff5-tjgs2
15:08:27	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedUpdateStatus	Operation cannot be fulfilled on horizontalpodautoscalers.autoscaling "isvc-sklearn-batcher-predictor": StorageError: invalid object, Code: 4, Key: /kubernetes.io/horizontalpodautoscalers/kserve-ci-e2e-test/isvc-sklearn-batcher-predictor, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 641b77d3-4cb4-4e92-8d53-307d6d567a6b, UID in object meta:
15:08:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-564b94bff5-tjgs2	Started	Started container storage-initializer
15:08:28	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-custom-predictor-564b94bff5-tjgs2	AddedInterface	Add eth0 [10.134.0.34/23] from ovn-kubernetes
15:08:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-564b94bff5-tjgs2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:08:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-564b94bff5-tjgs2	Created	Created container: storage-initializer
15:08:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-564b94bff5-tjgs2	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:93a3c88e5cf08b3e8030b29f6d098cd5406a6fd3ad769e6e697c2ae6303a7cab" already present on machine
15:08:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-564b94bff5-tjgs2	Created	Created container: agent
15:08:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-564b94bff5-tjgs2	Started	Started container kserve-container
15:08:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-564b94bff5-tjgs2	Created	Created container: kserve-container
15:08:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-564b94bff5-tjgs2	Started	Started container agent
15:08:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-564b94bff5-tjgs2	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1407" already present on machine
15:08:50 (x12)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-5d8576b666-c782k	Unhealthy	Readiness probe failed: dial tcp 10.134.0.33:8080: connect: connection refused
15:08:50 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-5d8576b666-c782k	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
15:09:12 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:09:12 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:09:42 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:09:42 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:09:45 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:09:45	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	InferenceServiceReady	InferenceService [isvc-sklearn-batcher-custom] is Ready
15:09:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-564b94bff5-tjgs2	Killing	Stopping container kserve-container
15:09:52	kserve-ci-e2e-test	deployment-controller	message-dumper-predictor	ScalingReplicaSet	Scaled up replica set message-dumper-predictor-7f66cccfb6 from 0 to 1
15:09:52	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	UpdateFailed	Failed to update status for InferenceService "message-dumper": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper": the object has been modified; please apply your changes to the latest version and try again
15:09:52	kserve-ci-e2e-test	replicaset-controller	message-dumper-predictor-7f66cccfb6	SuccessfulCreate	Created pod: message-dumper-predictor-7f66cccfb6-2ttq5
15:09:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-564b94bff5-tjgs2	Killing	Stopping container agent
15:09:53	kserve-ci-e2e-test	kubelet	message-dumper-predictor-7f66cccfb6-2ttq5	Pulled	Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 886ms (886ms including waiting). Image size: 14813193 bytes.
15:09:53	kserve-ci-e2e-test	multus	message-dumper-predictor-7f66cccfb6-2ttq5	AddedInterface	Add eth0 [10.133.0.28/23] from ovn-kubernetes
15:09:53	kserve-ci-e2e-test	kubelet	message-dumper-predictor-7f66cccfb6-2ttq5	Pulling	Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display"
15:09:54	kserve-ci-e2e-test	kubelet	message-dumper-predictor-7f66cccfb6-2ttq5	Unhealthy	Readiness probe failed: dial tcp 10.133.0.28:8080: connect: connection refused
15:09:54	kserve-ci-e2e-test	kubelet	message-dumper-predictor-7f66cccfb6-2ttq5	Started	Started container kserve-container
15:09:54	kserve-ci-e2e-test	kubelet	message-dumper-predictor-7f66cccfb6-2ttq5	Created	Created container: kserve-container
15:09:55	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	InferenceServiceReady	InferenceService [message-dumper] is Ready
15:09:55 (x8)	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:10:02	kserve-ci-e2e-test	deployment-controller	isvc-logger-predictor	ScalingReplicaSet	Scaled up replica set isvc-logger-predictor-7c7767ff77 from 0 to 1
15:10:03	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7c7767ff77-fcpnf	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:10:03	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7c7767ff77-fcpnf	Created	Created container: storage-initializer
15:10:03	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7c7767ff77-fcpnf	Started	Started container storage-initializer
15:10:03 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	UpdateFailed	Failed to update status for InferenceService "isvc-logger": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger": the object has been modified; please apply your changes to the latest version and try again
15:10:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger": the object has been modified; please apply your changes to the latest version and try again
15:10:03	kserve-ci-e2e-test	multus	isvc-logger-predictor-7c7767ff77-fcpnf	AddedInterface	Add eth0 [10.134.0.35/23] from ovn-kubernetes
15:10:03	kserve-ci-e2e-test	replicaset-controller	isvc-logger-predictor-7c7767ff77	SuccessfulCreate	Created pod: isvc-logger-predictor-7c7767ff77-fcpnf
15:10:07	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7c7767ff77-fcpnf	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1407" already present on machine
15:10:07	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7c7767ff77-fcpnf	Started	Started container kserve-container
15:10:07	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7c7767ff77-fcpnf	Created	Created container: agent
15:10:07	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7c7767ff77-fcpnf	Started	Started container agent
15:10:07	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7c7767ff77-fcpnf	Created	Created container: kserve-container
15:10:07	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7c7767ff77-fcpnf	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:93a3c88e5cf08b3e8030b29f6d098cd5406a6fd3ad769e6e697c2ae6303a7cab" already present on machine
15:10:15 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-564b94bff5-tjgs2	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
15:10:15 (x12)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-564b94bff5-tjgs2	Unhealthy	Readiness probe failed: dial tcp 10.134.0.34:5000: connect: connection refused
15:10:22 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:10:22 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:10:32 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:10:32 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:11:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:11:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:11:20 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:11:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	InferenceServiceReady	InferenceService [isvc-logger] is Ready
15:11:28	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7c7767ff77-fcpnf	Killing	Stopping container kserve-container
15:11:28	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-predictor-78c8d484d6 from 0 to 1
15:11:28	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-78c8d484d6-xghcj	Started	Started container storage-initializer
15:11:28	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-78c8d484d6-xghcj	Created	Created container: storage-initializer
15:11:28	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-78c8d484d6-xghcj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:11:28	kserve-ci-e2e-test	multus	isvc-lightgbm-predictor-78c8d484d6-xghcj	AddedInterface	Add eth0 [10.134.0.36/23] from ovn-kubernetes
15:11:28	kserve-ci-e2e-test	kubelet	message-dumper-predictor-7f66cccfb6-2ttq5	Killing	Stopping container kserve-container
15:11:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm": the object has been modified; please apply your changes to the latest version and try again
15:11:28	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7c7767ff77-fcpnf	Killing	Stopping container agent
15:11:28	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-predictor-78c8d484d6	SuccessfulCreate	Created pod: isvc-lightgbm-predictor-78c8d484d6-xghcj
15:11:33	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-78c8d484d6-xghcj	Pulling	Pulling image "kserve/lgbserver:latest"
15:11:40	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-78c8d484d6-xghcj	Started	Started container kserve-container
15:11:40	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-78c8d484d6-xghcj	Pulled	Successfully pulled image "kserve/lgbserver:latest" in 6.89s (6.89s including waiting). Image size: 606108943 bytes.
15:11:40	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-78c8d484d6-xghcj	Created	Created container: kserve-container
15:11:50 (x12)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7c7767ff77-fcpnf	Unhealthy	Readiness probe failed: dial tcp 10.134.0.35:8080: connect: connection refused
15:11:50 (x11)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7c7767ff77-fcpnf	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
15:12:13 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:12:13 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:12:58 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:12:58 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:13:01 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:13:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	InferenceServiceReady	InferenceService [isvc-lightgbm] is Ready
15:13:08	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-78c8d484d6-xghcj	Killing	Stopping container kserve-container
15:13:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-runtime": the object has been modified; please apply your changes to the latest version and try again
15:13:08 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-runtime": the object has been modified; please apply your changes to the latest version and try again
15:13:08	kserve-ci-e2e-test	multus	isvc-lightgbm-runtime-predictor-64984c7cb-dwxrv	AddedInterface	Add eth0 [10.134.0.37/23] from ovn-kubernetes
15:13:08	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-64984c7cb-dwxrv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:13:08	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-runtime-predictor-64984c7cb from 0 to 1
15:13:08	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-runtime-predictor-64984c7cb	SuccessfulCreate	Created pod: isvc-lightgbm-runtime-predictor-64984c7cb-dwxrv
15:13:09	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-64984c7cb-dwxrv	Created	Created container: storage-initializer
15:13:09	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-64984c7cb-dwxrv	Started	Started container storage-initializer
15:13:11 (x10)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-78c8d484d6-xghcj	Unhealthy	Readiness probe failed: dial tcp 10.134.0.36:8080: connect: connection refused
15:13:13	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-64984c7cb-dwxrv	Created	Created container: kserve-container
15:13:13	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-64984c7cb-dwxrv	Started	Started container kserve-container
15:13:13	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-64984c7cb-dwxrv	Pulled	Container image "kserve/lgbserver:latest" already present on machine
15:13:53 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:13:53 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:14:25 (x9)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-64984c7cb-dwxrv	Unhealthy	Readiness probe failed: dial tcp 10.134.0.37:8080: connect: connection refused
15:14:35 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:14:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-runtime] is Ready
15:14:38	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-runtime-predictor-8d44c64dc from 0 to 1
15:14:38	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-64984c7cb-dwxrv	Killing	Stopping container kserve-container
15:14:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:14:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:14:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
15:14:39	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-runtime-predictor-8d44c64dc	SuccessfulCreate	Created pod: isvc-lightgbm-v2-runtime-predictor-8d44c64dc-7kmd5
15:14:39	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-runtime-predictor-8d44c64dc-7kmd5	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
15:14:39	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8d44c64dc-7kmd5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:14:39	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8d44c64dc-7kmd5	Created	Created container: storage-initializer
15:14:39	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8d44c64dc-7kmd5	Started	Started container storage-initializer
15:14:39 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
15:14:43	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8d44c64dc-7kmd5	Pulling	Pulling image "docker.io/seldonio/mlserver:1.7.1"
15:17:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8d44c64dc-7kmd5	Pulled	Successfully pulled image "docker.io/seldonio/mlserver:1.7.1" in 2m16.745s (2m16.745s including waiting). Image size: 10890461297 bytes.
15:17:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8d44c64dc-7kmd5	Created	Created container: kserve-container
15:17:00	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8d44c64dc-7kmd5	Started	Started container kserve-container
15:17:24 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:17:24 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:17:32	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-runtime] is Ready
15:17:32 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:17:39	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-bcss7	Started	Started container storage-initializer
15:17:39	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:17:39	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-kserve-predictor-68fc4f5b49	SuccessfulCreate	Created pod: isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-bcss7
15:17:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
15:17:39	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-bcss7	Created	Created container: storage-initializer
15:17:39	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8d44c64dc-7kmd5	Killing	Stopping container kserve-container
15:17:39	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:17:39	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-bcss7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:17:39	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-kserve-predictor-68fc4f5b49 from 0 to 1
15:17:39	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-bcss7	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
15:17:43	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-bcss7	Started	Started container kserve-container
15:17:43	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-bcss7	Created	Created container: kserve-container
15:17:43	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-bcss7	Pulled	Container image "kserve/lgbserver:latest" already present on machine
15:17:45 (x2)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-bcss7	Unhealthy	Readiness probe failed: dial tcp 10.134.0.39:8080: connect: connection refused
15:17:54	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:17:54	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:17:55 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:17:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-kserve] is Ready
15:17:59	kserve-ci-e2e-test	deployment-controller	isvc-mlflow-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-mlflow-v2-runtime-predictor-5497fc5f68 from 0 to 1
15:17:59	kserve-ci-e2e-test	multus	isvc-mlflow-v2-runtime-predictor-5497fc5f68-lxbwj	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
15:17:59	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5497fc5f68-lxbwj	Created	Created container: storage-initializer
15:17:59	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-bcss7	Killing	Stopping container kserve-container
15:17:59	kserve-ci-e2e-test	replicaset-controller	isvc-mlflow-v2-runtime-predictor-5497fc5f68	SuccessfulCreate	Created pod: isvc-mlflow-v2-runtime-predictor-5497fc5f68-lxbwj
15:17:59 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-mlflow-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-mlflow-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
15:17:59	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5497fc5f68-lxbwj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:17:59	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-mlflow-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
15:18:00	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5497fc5f68-lxbwj	Started	Started container storage-initializer
15:18:04	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5497fc5f68-lxbwj	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
15:18:04	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5497fc5f68-lxbwj	Created	Created container: kserve-container
15:18:04	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5497fc5f68-lxbwj	Started	Started container kserve-container
15:18:29 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:18:29 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:18:36	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	InferenceServiceReady	InferenceService [isvc-mlflow-v2-runtime] is Ready
15:18:36 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:18:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-mcp": the object has been modified; please apply your changes to the latest version and try again
15:18:39	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-mcp-predictor-65975766bf	SuccessfulCreate	Created pod: isvc-sklearn-mcp-predictor-65975766bf-cz4hx
15:18:39	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5497fc5f68-lxbwj	Killing	Stopping container kserve-container
15:18:39	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-mcp-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-mcp-predictor-65975766bf from 0 to 1
15:18:39 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-mcp": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-mcp": the object has been modified; please apply your changes to the latest version and try again
15:18:40	kserve-ci-e2e-test	multus	isvc-sklearn-mcp-predictor-65975766bf-cz4hx	AddedInterface	Add eth0 [10.134.0.41/23] from ovn-kubernetes
15:18:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-65975766bf-cz4hx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:18:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-65975766bf-cz4hx	Created	Created container: storage-initializer
15:18:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-65975766bf-cz4hx	Started	Started container storage-initializer
15:18:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-65975766bf-cz4hx	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1407" already present on machine
15:18:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-65975766bf-cz4hx	Created	Created container: kserve-container
15:18:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-65975766bf-cz4hx	Started	Started container kserve-container
15:18:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-65975766bf-cz4hx	Pulling	Pulling image "quay.io/opendatahub/kserve-agent:latest"
15:18:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-65975766bf-cz4hx	Started	Started container kserve-agent
15:18:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-65975766bf-cz4hx	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent:latest" in 3.37s (3.37s including waiting). Image size: 237663781 bytes.
15:18:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-65975766bf-cz4hx	Created	Created container: kserve-agent
15:19:49 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:19:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	InferenceServiceReady	InferenceService [isvc-sklearn-mcp] is Ready
15:19:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-65975766bf-cz4hx	Killing	Stopping container kserve-agent
15:19:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-65975766bf-cz4hx	Killing	Stopping container kserve-container
15:19:59	kserve-ci-e2e-test	deployment-controller	isvc-paddle-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-predictor-7dddcb4bd4 from 0 to 1
15:19:59	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	UpdateFailed	Failed to update status for InferenceService "isvc-paddle": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle": the object has been modified; please apply your changes to the latest version and try again
15:19:59	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-predictor-7dddcb4bd4	SuccessfulCreate	Created pod: isvc-paddle-predictor-7dddcb4bd4-h6xds
15:20:00	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-7dddcb4bd4-h6xds	Started	Started container storage-initializer
15:20:00	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-7dddcb4bd4-h6xds	Created	Created container: storage-initializer
15:20:00	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-7dddcb4bd4-h6xds	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:20:00	kserve-ci-e2e-test	multus	isvc-paddle-predictor-7dddcb4bd4-h6xds	AddedInterface	Add eth0 [10.134.0.42/23] from ovn-kubernetes
15:20:05	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-7dddcb4bd4-h6xds	Pulling	Pulling image "kserve/paddleserver:latest"
15:20:17	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-7dddcb4bd4-h6xds	Pulled	Successfully pulled image "kserve/paddleserver:latest" in 12.193s (12.193s including waiting). Image size: 1162639611 bytes.
15:20:17	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-7dddcb4bd4-h6xds	Created	Created container: kserve-container
15:20:17	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-7dddcb4bd4-h6xds	Started	Started container kserve-container
15:20:19 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-65975766bf-cz4hx	Unhealthy	Readiness probe failed: Get "http://10.134.0.41:8080/v1/models/isvc-sklearn-mcp": dial tcp 10.134.0.41:8080: connect: connection refused
15:20:29 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:20:29 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:20:30	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-65975766bf-cz4hx	Unhealthy	Readiness probe failed: dial tcp 10.134.0.41:9081: i/o timeout
15:20:30	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-65975766bf-cz4hx	Unhealthy	Readiness probe failed: Get "http://10.134.0.41:8080/v1/models/isvc-sklearn-mcp": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
15:20:59 (x6)	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-7dddcb4bd4-h6xds	Unhealthy	Readiness probe failed: dial tcp 10.134.0.42:8080: connect: connection refused
15:20:59 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:20:59 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:21:09	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	InferenceServiceReady	InferenceService [isvc-paddle] is Ready
15:21:09 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:21:11	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-runtime-predictor-86b49c4466	SuccessfulCreate	Created pod: isvc-paddle-runtime-predictor-86b49c4466-xklbk
15:21:11	kserve-ci-e2e-test	multus	isvc-paddle-runtime-predictor-86b49c4466-xklbk	AddedInterface	Add eth0 [10.132.0.18/23] from ovn-kubernetes
15:21:11	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-86b49c4466-xklbk	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a"
15:21:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-runtime": the object has been modified; please apply your changes to the latest version and try again
15:21:11	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-7dddcb4bd4-h6xds	Killing	Stopping container kserve-container
15:21:11	kserve-ci-e2e-test	deployment-controller	isvc-paddle-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-runtime-predictor-86b49c4466 from 0 to 1
15:21:16	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-86b49c4466-xklbk	Created	Created container: storage-initializer
15:21:16	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-86b49c4466-xklbk	Started	Started container storage-initializer
15:21:16	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-86b49c4466-xklbk	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" in 5.065s (5.065s including waiting). Image size: 300960724 bytes.
15:21:21	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-86b49c4466-xklbk	Pulling	Pulling image "kserve/paddleserver:latest"
15:21:33	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-86b49c4466-xklbk	Created	Created container: kserve-container
15:21:33	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-86b49c4466-xklbk	Started	Started container kserve-container
15:21:33	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-86b49c4466-xklbk	Pulled	Successfully pulled image "kserve/paddleserver:latest" in 11.91s (11.91s including waiting). Image size: 1162639611 bytes.
15:21:41 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:21:41 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:22:05 (x5)	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-86b49c4466-xklbk	Unhealthy	Readiness probe failed: dial tcp 10.132.0.18:8080: connect: connection refused
15:22:11 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:22:11 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:22:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	InferenceServiceReady	InferenceService [isvc-paddle-runtime] is Ready
15:22:15 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:22:22	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-86b49c4466-xklbk	Killing	Stopping container kserve-container
15:22:22	kserve-ci-e2e-test	deployment-controller	isvc-paddle-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-v2-kserve-predictor-7c6bd76f7b from 0 to 1
15:22:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
15:22:22	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-v2-kserve-predictor-7c6bd76f7b	SuccessfulCreate	Created pod: isvc-paddle-v2-kserve-predictor-7c6bd76f7b-bc22l
15:22:22 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
15:22:23	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7c6bd76f7b-bc22l	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:22:23	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7c6bd76f7b-bc22l	Started	Started container storage-initializer
15:22:23	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7c6bd76f7b-bc22l	Created	Created container: storage-initializer
15:22:23	kserve-ci-e2e-test	multus	isvc-paddle-v2-kserve-predictor-7c6bd76f7b-bc22l	AddedInterface	Add eth0 [10.132.0.19/23] from ovn-kubernetes
15:22:28	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7c6bd76f7b-bc22l	Pulled	Container image "kserve/paddleserver:latest" already present on machine
15:22:28	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7c6bd76f7b-bc22l	Created	Created container: kserve-container
15:22:28	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7c6bd76f7b-bc22l	Started	Started container kserve-container
15:22:37	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:22:37	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:23:00 (x5)	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7c6bd76f7b-bc22l	Unhealthy	Readiness probe failed: dial tcp 10.132.0.19:8080: connect: connection refused
15:23:07 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:23:07 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:23:10 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:23:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	InferenceServiceReady	InferenceService [isvc-paddle-v2-kserve] is Ready
15:23:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	UpdateFailed	Failed to update status for InferenceService "isvc-pmml": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml": the object has been modified; please apply your changes to the latest version and try again
15:23:14	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-predictor-5584ffd8c9	SuccessfulCreate	Created pod: isvc-pmml-predictor-5584ffd8c9-ft682
15:23:14	kserve-ci-e2e-test	deployment-controller	isvc-pmml-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-predictor-5584ffd8c9 from 0 to 1
15:23:14	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7c6bd76f7b-bc22l	Killing	Stopping container kserve-container
15:23:15	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-5584ffd8c9-ft682	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:23:15	kserve-ci-e2e-test	multus	isvc-pmml-predictor-5584ffd8c9-ft682	AddedInterface	Add eth0 [10.132.0.20/23] from ovn-kubernetes
15:23:15	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-5584ffd8c9-ft682	Created	Created container: storage-initializer
15:23:15	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-5584ffd8c9-ft682	Started	Started container storage-initializer
15:23:19	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-5584ffd8c9-ft682	Pulling	Pulling image "kserve/pmmlserver:latest"
15:23:26	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-5584ffd8c9-ft682	Started	Started container kserve-container
15:23:26	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-5584ffd8c9-ft682	Created	Created container: kserve-container
15:23:26	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-5584ffd8c9-ft682	Pulled	Successfully pulled image "kserve/pmmlserver:latest" in 6.875s (6.875s including waiting). Image size: 800715639 bytes.
15:23:44 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:23:44 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:24:44 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:24:44 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:24:46	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	InferenceServiceReady	InferenceService [isvc-pmml] is Ready
15:24:46 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:24:55	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-5584ffd8c9-ft682	Killing	Stopping container kserve-container
15:24:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-runtime": the object has been modified; please apply your changes to the latest version and try again
15:24:55	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-runtime-predictor-7576f6b69f	SuccessfulCreate	Created pod: isvc-pmml-runtime-predictor-7576f6b69f-zt4tv
15:24:55	kserve-ci-e2e-test	deployment-controller	isvc-pmml-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-runtime-predictor-7576f6b69f from 0 to 1
15:24:55 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-runtime": the object has been modified; please apply your changes to the latest version and try again
15:24:56	kserve-ci-e2e-test	multus	isvc-pmml-runtime-predictor-7576f6b69f-zt4tv	AddedInterface	Add eth0 [10.132.0.21/23] from ovn-kubernetes
15:24:56	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-7576f6b69f-zt4tv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:24:56	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-7576f6b69f-zt4tv	Created	Created container: storage-initializer
15:24:56	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-7576f6b69f-zt4tv	Started	Started container storage-initializer
15:24:56 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-5584ffd8c9-ft682	Unhealthy	Readiness probe failed: dial tcp 10.132.0.20:8080: connect: connection refused
15:25:01	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-7576f6b69f-zt4tv	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
15:25:01	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-7576f6b69f-zt4tv	Created	Created container: kserve-container
15:25:01	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-7576f6b69f-zt4tv	Started	Started container kserve-container
15:25:40 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:25:40 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:26:14 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-7576f6b69f-zt4tv	Unhealthy	Readiness probe failed: dial tcp 10.132.0.21:8080: connect: connection refused
15:26:24 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:26:24	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	InferenceServiceReady	InferenceService [isvc-pmml-runtime] is Ready
15:26:25 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:26:25 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:26:26	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-v2-kserve-predictor-75b87ff64c	SuccessfulCreate	Created pod: isvc-pmml-v2-kserve-predictor-75b87ff64c-24hvt
15:26:26	kserve-ci-e2e-test	deployment-controller	isvc-pmml-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-v2-kserve-predictor-75b87ff64c from 0 to 1
15:26:26	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
15:26:26	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-7576f6b69f-zt4tv	Killing	Stopping container kserve-container
15:26:27	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-75b87ff64c-24hvt	Started	Started container storage-initializer
15:26:27	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-75b87ff64c-24hvt	Created	Created container: storage-initializer
15:26:27	kserve-ci-e2e-test	multus	isvc-pmml-v2-kserve-predictor-75b87ff64c-24hvt	AddedInterface	Add eth0 [10.132.0.22/23] from ovn-kubernetes
15:26:27	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-75b87ff64c-24hvt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:26:31	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-75b87ff64c-24hvt	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
15:26:31	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-75b87ff64c-24hvt	Created	Created container: kserve-container
15:26:31	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-75b87ff64c-24hvt	Started	Started container kserve-container
15:27:11 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:27:11 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:27:42 (x9)	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-75b87ff64c-24hvt	Unhealthy	Readiness probe failed: dial tcp 10.132.0.22:8080: connect: connection refused
15:27:52 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:27:52	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	InferenceServiceReady	InferenceService [isvc-pmml-v2-kserve] is Ready
15:27:56 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:27:56 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:27:57	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-75b87ff64c-24hvt	Killing	Stopping container kserve-container
15:27:58	kserve-ci-e2e-test	kubelet	isvc-primary-21dbf3-predictor-6bd6957cff-swgrv	Started	Started container storage-initializer
15:27:58	kserve-ci-e2e-test	kubelet	isvc-primary-21dbf3-predictor-6bd6957cff-swgrv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:27:58 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-21dbf3	UpdateFailed	Failed to update status for InferenceService "isvc-primary-21dbf3": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-21dbf3": the object has been modified; please apply your changes to the latest version and try again
15:27:58	kserve-ci-e2e-test	deployment-controller	isvc-primary-21dbf3-predictor	ScalingReplicaSet	Scaled up replica set isvc-primary-21dbf3-predictor-6bd6957cff from 0 to 1
15:27:58	kserve-ci-e2e-test	replicaset-controller	isvc-primary-21dbf3-predictor-6bd6957cff	SuccessfulCreate	Created pod: isvc-primary-21dbf3-predictor-6bd6957cff-swgrv
15:27:58 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-21dbf3	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-21dbf3": the object has been modified; please apply your changes to the latest version and try again
15:27:58	kserve-ci-e2e-test	kubelet	isvc-primary-21dbf3-predictor-6bd6957cff-swgrv	Created	Created container: storage-initializer
15:27:58	kserve-ci-e2e-test	multus	isvc-primary-21dbf3-predictor-6bd6957cff-swgrv	AddedInterface	Add eth0 [10.134.0.43/23] from ovn-kubernetes
15:28:02	kserve-ci-e2e-test	kubelet	isvc-primary-21dbf3-predictor-6bd6957cff-swgrv	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1407" already present on machine
15:28:02	kserve-ci-e2e-test	kubelet	isvc-primary-21dbf3-predictor-6bd6957cff-swgrv	Created	Created container: kserve-container
15:28:02	kserve-ci-e2e-test	kubelet	isvc-primary-21dbf3-predictor-6bd6957cff-swgrv	Started	Started container kserve-container
15:28:13	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-21dbf3-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:28:13	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-21dbf3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:29:04 (x8)	kserve-ci-e2e-test	kubelet	isvc-primary-21dbf3-predictor-6bd6957cff-swgrv	Unhealthy	Readiness probe failed: dial tcp 10.134.0.43:8080: connect: connection refused
15:29:13	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-21dbf3	InferenceServiceReady	InferenceService [isvc-primary-21dbf3] is Ready
15:29:13 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-21dbf3	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:29:18 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-21dbf3	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-21dbf3": the object has been modified; please apply your changes to the latest version and try again
15:29:18	kserve-ci-e2e-test	multus	isvc-secondary-21dbf3-predictor-78d7c59b5d-rfdhs	AddedInterface	Add eth0 [10.134.0.44/23] from ovn-kubernetes
15:29:18	kserve-ci-e2e-test	replicaset-controller	isvc-secondary-21dbf3-predictor-78d7c59b5d	SuccessfulCreate	Created pod: isvc-secondary-21dbf3-predictor-78d7c59b5d-rfdhs
15:29:18	kserve-ci-e2e-test	deployment-controller	isvc-secondary-21dbf3-predictor	ScalingReplicaSet	Scaled up replica set isvc-secondary-21dbf3-predictor-78d7c59b5d from 0 to 1
15:29:18 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-21dbf3	UpdateFailed	Failed to update status for InferenceService "isvc-secondary-21dbf3": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-21dbf3": the object has been modified; please apply your changes to the latest version and try again
15:29:25 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-21dbf3-predictor-78d7c59b5d-rfdhs	Started	Started container storage-initializer
15:29:25 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-21dbf3-predictor-78d7c59b5d-rfdhs	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:29:25 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-21dbf3-predictor-78d7c59b5d-rfdhs	Created	Created container: storage-initializer
15:29:28 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-21dbf3-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:29:28 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-21dbf3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:29:31	kserve-ci-e2e-test	kubelet	isvc-secondary-21dbf3-predictor-78d7c59b5d-rfdhs	BackOff	Back-off restarting failed container storage-initializer in pod isvc-secondary-21dbf3-predictor-78d7c59b5d-rfdhs_kserve-ci-e2e-test(1df531f6-39cd-4702-b47f-a3d6fe125f73)
15:29:31 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-21dbf3	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:29:33	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-21dbf3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:29:33	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-21dbf3-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:29:36 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-db90c5	UpdateFailed	Failed to update status for InferenceService "isvc-init-fail-db90c5": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-db90c5": the object has been modified; please apply your changes to the latest version and try again
15:29:36	kserve-ci-e2e-test	replicaset-controller	isvc-init-fail-db90c5-predictor-cd8bb7b7d	SuccessfulCreate	Created pod: isvc-init-fail-db90c5-predictor-cd8bb7b7d-ldt9w
15:29:36	kserve-ci-e2e-test	multus	isvc-init-fail-db90c5-predictor-cd8bb7b7d-ldt9w	AddedInterface	Add eth0 [10.134.0.45/23] from ovn-kubernetes
15:29:36	kserve-ci-e2e-test	deployment-controller	isvc-init-fail-db90c5-predictor	ScalingReplicaSet	Scaled up replica set isvc-init-fail-db90c5-predictor-cd8bb7b7d from 0 to 1
15:29:36 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-db90c5	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-db90c5": the object has been modified; please apply your changes to the latest version and try again
15:29:36	kserve-ci-e2e-test	kubelet	isvc-primary-21dbf3-predictor-6bd6957cff-swgrv	Killing	Stopping container kserve-container
15:29:44 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-db90c5-predictor-cd8bb7b7d-ldt9w	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:29:44 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-db90c5-predictor-cd8bb7b7d-ldt9w	Started	Started container storage-initializer
15:29:44 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-db90c5-predictor-cd8bb7b7d-ldt9w	Created	Created container: storage-initializer
15:29:46	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn": the object has been modified; please apply your changes to the latest version and try again
15:29:46	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-85bccb8945-hkrrp	Started	Started container storage-initializer
15:29:46 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-db90c5	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:29:46	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-predictor-85bccb8945	SuccessfulCreate	Created pod: isvc-predictive-sklearn-predictor-85bccb8945-hkrrp
15:29:46	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-predictor-85bccb8945 from 0 to 1
15:29:46	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-85bccb8945-hkrrp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:29:46	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-predictor-85bccb8945-hkrrp	AddedInterface	Add eth0 [10.132.0.23/23] from ovn-kubernetes
15:29:46	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-85bccb8945-hkrrp	Created	Created container: storage-initializer
15:29:46	kserve-ci-e2e-test	kubelet	isvc-init-fail-db90c5-predictor-cd8bb7b7d-ldt9w	BackOff	Back-off restarting failed container storage-initializer in pod isvc-init-fail-db90c5-predictor-cd8bb7b7d-ldt9w_kserve-ci-e2e-test(9dbe00f2-9474-4aaa-b798-ef50c2b680d2)
15:29:50	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-85bccb8945-hkrrp	Pulling	Pulling image "kserve/predictiveserver:latest"
15:30:12	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-85bccb8945-hkrrp	Pulled	Successfully pulled image "kserve/predictiveserver:latest" in 21.666s (21.666s including waiting). Image size: 2312633199 bytes.
15:30:12	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-85bccb8945-hkrrp	Created	Created container: kserve-container
15:30:12	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-85bccb8945-hkrrp	Started	Started container kserve-container
15:30:31 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:30:31 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:31:24 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-85bccb8945-hkrrp	Unhealthy	Readiness probe failed: dial tcp 10.132.0.23:8080: connect: connection refused
15:31:31 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:31:31 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:31:34 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:31:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	InferenceServiceReady	InferenceService [isvc-predictive-sklearn] is Ready
15:31:36	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost": the object has been modified; please apply your changes to the latest version and try again
15:31:36	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-85bccb8945-hkrrp	Killing	Stopping container kserve-container
15:31:36	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-predictor-794b54b9b4	SuccessfulCreate	Created pod: isvc-predictive-xgboost-predictor-794b54b9b4-hxq4v
15:31:36	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-predictor-794b54b9b4 from 0 to 1
15:31:37	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-794b54b9b4-hxq4v	Created	Created container: storage-initializer
15:31:37	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-794b54b9b4-hxq4v	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:31:37	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-predictor-794b54b9b4-hxq4v	AddedInterface	Add eth0 [10.132.0.24/23] from ovn-kubernetes
15:31:37	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-794b54b9b4-hxq4v	Started	Started container storage-initializer
15:31:41	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-794b54b9b4-hxq4v	Started	Started container kserve-container
15:31:41	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-794b54b9b4-hxq4v	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
15:31:41	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-794b54b9b4-hxq4v	Created	Created container: kserve-container
15:31:51	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:31:51	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:32:43 (x8)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-794b54b9b4-hxq4v	Unhealthy	Readiness probe failed: dial tcp 10.132.0.24:8080: connect: connection refused
15:32:51 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:32:51 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:32:53 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:32:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	InferenceServiceReady	InferenceService [isvc-predictive-xgboost] is Ready
15:32:56	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-794b54b9b4-hxq4v	Killing	Stopping container kserve-container
15:32:56	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-predictor-669896799c from 0 to 1
15:32:56	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-predictor-669896799c	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-predictor-669896799c-zvdk7
15:32:57 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm": the object has been modified; please apply your changes to the latest version and try again
15:32:57	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm": the object has been modified; please apply your changes to the latest version and try again
15:32:57	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-predictor-669896799c-zvdk7	AddedInterface	Add eth0 [10.132.0.25/23] from ovn-kubernetes
15:32:57	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-669896799c-zvdk7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:32:57	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-669896799c-zvdk7	Started	Started container storage-initializer
15:32:57	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-669896799c-zvdk7	Created	Created container: storage-initializer
15:33:01	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-669896799c-zvdk7	Created	Created container: kserve-container
15:33:01	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-669896799c-zvdk7	Started	Started container kserve-container
15:33:01	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-669896799c-zvdk7	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
15:33:11	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:33:11	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:34:22 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:34:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm] is Ready
15:34:27	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-v2-predictor-6fd9c49f4f from 0 to 1
15:34:27	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
15:34:27 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:34:27 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:34:27	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-669896799c-zvdk7	Killing	Stopping container kserve-container
15:34:27	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-v2-predictor-6fd9c49f4f	SuccessfulCreate	Created pod: isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-68rw9
15:34:28	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-68rw9	Created	Created container: storage-initializer
15:34:28	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-68rw9	AddedInterface	Add eth0 [10.132.0.26/23] from ovn-kubernetes
15:34:28	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-68rw9	Started	Started container storage-initializer
15:34:28	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-68rw9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:34:32	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-68rw9	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
15:34:32 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-669896799c-zvdk7	Unhealthy	Readiness probe failed: dial tcp 10.132.0.25:8080: connect: connection refused
15:34:32	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-68rw9	Created	Created container: kserve-container
15:34:32	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-68rw9	Started	Started container kserve-container
15:35:12 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:35:12 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:35:34 (x4)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-68rw9	Unhealthy	Readiness probe failed: Get "http://10.132.0.26:8080/v2/models/isvc-predictive-sklearn-v2/ready": dial tcp 10.132.0.26:8080: connect: connection refused
15:35:42 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:35:42 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:35:44 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:35:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	InferenceServiceReady	InferenceService [isvc-predictive-sklearn-v2] is Ready
15:35:47	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-68rw9	Killing	Stopping container kserve-container
15:35:47	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-v2-predictor-67fd65d6cb	SuccessfulCreate	Created pod: isvc-predictive-xgboost-v2-predictor-67fd65d6cb-vpbbk
15:35:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
15:35:47 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
15:35:47	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-v2-predictor-67fd65d6cb from 0 to 1
15:35:48	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-v2-predictor-67fd65d6cb-vpbbk	AddedInterface	Add eth0 [10.132.0.27/23] from ovn-kubernetes
15:35:48	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-67fd65d6cb-vpbbk	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:35:48	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-67fd65d6cb-vpbbk	Created	Created container: storage-initializer
15:35:48	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-67fd65d6cb-vpbbk	Started	Started container storage-initializer
15:35:52	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-67fd65d6cb-vpbbk	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
15:35:52	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-67fd65d6cb-vpbbk	Started	Started container kserve-container
15:35:52	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-67fd65d6cb-vpbbk	Created	Created container: kserve-container
15:36:17 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:36:17 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:36:54 (x4)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-67fd65d6cb-vpbbk	Unhealthy	Readiness probe failed: Get "http://10.132.0.27:8080/v2/models/isvc-predictive-xgboost-v2/ready": dial tcp 10.132.0.27:8080: connect: connection refused
15:37:02 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:37:02 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:37:04 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:37:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	InferenceServiceReady	InferenceService [isvc-predictive-xgboost-v2] is Ready
15:37:08	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-65df45cf79-sbblm	Created	Created container: storage-initializer
15:37:08	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-v2-predictor-65df45cf79-sbblm	AddedInterface	Add eth0 [10.132.0.28/23] from ovn-kubernetes
15:37:08	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-65df45cf79-sbblm	Started	Started container storage-initializer
15:37:08	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-v2-predictor-65df45cf79	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-v2-predictor-65df45cf79-sbblm
15:37:08	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-65df45cf79-sbblm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:37:08	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-67fd65d6cb-vpbbk	Killing	Stopping container kserve-container
15:37:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm-v2": the object has been modified; please apply your changes to the latest version and try again
15:37:08 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm-v2": the object has been modified; please apply your changes to the latest version and try again
15:37:08	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-v2-predictor-65df45cf79 from 0 to 1
15:37:13	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-65df45cf79-sbblm	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
15:37:13	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-65df45cf79-sbblm	Started	Started container kserve-container
15:37:13	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-65df45cf79-sbblm	Created	Created container: kserve-container
15:37:38 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:37:38 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:38:25 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-65df45cf79-sbblm	Unhealthy	Readiness probe failed: Get "http://10.132.0.28:8080/v2/models/isvc-predictive-lightgbm-v2/ready": dial tcp 10.132.0.28:8080: connect: connection refused
15:38:35 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:38:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm-v2] is Ready
15:38:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
15:38:38 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:38:38	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-scheduler-predictor-74bcbf56c6	SuccessfulCreate	Created pod: isvc-sklearn-scheduler-predictor-74bcbf56c6-6jfxk
15:38:38	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-65df45cf79-sbblm	Killing	Stopping container kserve-container
15:38:38	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-scheduler-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-scheduler-predictor-74bcbf56c6 from 0 to 1
15:38:38 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:38:38 (x6)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:38:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-scheduler": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
15:38:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-74df74d548-5559r	Started	Started container storage-initializer
15:38:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-74df74d548-5559r	Created	Created container: storage-initializer
15:38:40	kserve-ci-e2e-test	multus	isvc-sklearn-predictor-74df74d548-5559r	AddedInterface	Add eth0 [10.134.0.46/23] from ovn-kubernetes
15:38:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-74df74d548-5559r	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:38:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn": the object has been modified; please apply your changes to the latest version and try again
15:38:40	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-predictor-74df74d548 from 0 to 1
15:38:40	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-predictor-74df74d548	SuccessfulCreate	Created pod: isvc-sklearn-predictor-74df74d548-5559r
15:38:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-74df74d548-5559r	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1407" already present on machine
15:38:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-74df74d548-5559r	Started	Started container kserve-container
15:38:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-74df74d548-5559r	Created	Created container: kserve-container
15:38:55	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:38:55	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:39:47 (x8)	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-74df74d548-5559r	Unhealthy	Readiness probe failed: dial tcp 10.134.0.46:8080: connect: connection refused
15:39:55 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:39:55 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:39:57 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:39:57	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	InferenceServiceReady	InferenceService [isvc-sklearn] is Ready
15:40:00	kserve-ci-e2e-test	replicaset-controller	sklearn-v2-mlserver-predictor-695d5f5568	SuccessfulCreate	Created pod: sklearn-v2-mlserver-predictor-695d5f5568-krhvs
15:40:00	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "sklearn-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
15:40:00 (x2)	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "sklearn-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "sklearn-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
15:40:00	kserve-ci-e2e-test	deployment-controller	sklearn-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set sklearn-v2-mlserver-predictor-695d5f5568 from 0 to 1
15:40:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-74df74d548-5559r	Killing	Stopping container kserve-container
15:40:01	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-695d5f5568-krhvs	Created	Created container: storage-initializer
15:40:01	kserve-ci-e2e-test	multus	sklearn-v2-mlserver-predictor-695d5f5568-krhvs	AddedInterface	Add eth0 [10.134.0.47/23] from ovn-kubernetes
15:40:01	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-695d5f5568-krhvs	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:40:01	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-695d5f5568-krhvs	Started	Started container storage-initializer
15:40:05	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-695d5f5568-krhvs	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
15:40:05	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-695d5f5568-krhvs	Created	Created container: kserve-container
15:40:05	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-695d5f5568-krhvs	Started	Started container kserve-container
15:40:37	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-695d5f5568-krhvs	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
15:40:45 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:40:45 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:40:47 (x14)	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:40:47	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	InferenceServiceReady	InferenceService [sklearn-v2-mlserver] is Ready
15:40:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-runtime": the object has been modified; please apply your changes to the latest version and try again
15:40:50	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-runtime-predictor-9c688ddc4 from 0 to 1
15:40:50	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-695d5f5568-krhvs	Killing	Stopping container kserve-container
15:40:50	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-runtime-predictor-9c688ddc4	SuccessfulCreate	Created pod: isvc-sklearn-runtime-predictor-9c688ddc4-nlk27
15:40:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-9c688ddc4-nlk27	Started	Started container storage-initializer
15:40:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-9c688ddc4-nlk27	Created	Created container: storage-initializer
15:40:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-9c688ddc4-nlk27	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:40:51	kserve-ci-e2e-test	multus	isvc-sklearn-runtime-predictor-9c688ddc4-nlk27	AddedInterface	Add eth0 [10.134.0.48/23] from ovn-kubernetes
15:40:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-9c688ddc4-nlk27	Started	Started container kserve-container
15:40:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-9c688ddc4-nlk27	Created	Created container: kserve-container
15:40:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-9c688ddc4-nlk27	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1407" already present on machine
15:40:57	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-695d5f5568-krhvs	Unhealthy	Readiness probe failed: Get "http://10.134.0.47:8080/v2/models/sklearn-v2-mlserver/ready": dial tcp 10.134.0.47:8080: connect: connection refused
15:41:09 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-9c688ddc4-nlk27	Unhealthy	Readiness probe failed: dial tcp 10.134.0.48:8080: connect: connection refused
15:41:19 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:41:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-runtime] is Ready
15:41:20 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:41:20 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:41:27	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-runtime-predictor-7dd9b85c64 from 0 to 1
15:41:27	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-runtime-predictor-7dd9b85c64	SuccessfulCreate	Created pod: isvc-sklearn-v2-runtime-predictor-7dd9b85c64-c8wt2
15:41:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-9c688ddc4-nlk27	Killing	Stopping container kserve-container
15:41:28	kserve-ci-e2e-test	multus	isvc-sklearn-v2-runtime-predictor-7dd9b85c64-c8wt2	AddedInterface	Add eth0 [10.134.0.49/23] from ovn-kubernetes
15:41:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-7dd9b85c64-c8wt2	Created	Created container: storage-initializer
15:41:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-7dd9b85c64-c8wt2	Started	Started container storage-initializer
15:41:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-7dd9b85c64-c8wt2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:41:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
15:41:28 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
15:41:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-7dd9b85c64-c8wt2	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
15:41:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-7dd9b85c64-c8wt2	Started	Started container kserve-container
15:41:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-7dd9b85c64-c8wt2	Created	Created container: kserve-container
15:42:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-7dd9b85c64-c8wt2	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
15:42:13 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:42:13 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:42:14 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:42:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-v2-runtime] is Ready
15:42:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-84556fcc44-2smfh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:42:18	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-predictor-84556fcc44 from 0 to 1
15:42:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
15:42:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-7dd9b85c64-c8wt2	Killing	Stopping container kserve-container
15:42:18	kserve-ci-e2e-test	multus	isvc-sklearn-v2-predictor-84556fcc44-2smfh	AddedInterface	Add eth0 [10.134.0.50/23] from ovn-kubernetes
15:42:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-84556fcc44-2smfh	Created	Created container: storage-initializer
15:42:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-84556fcc44-2smfh	Started	Started container storage-initializer
15:42:18	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-predictor-84556fcc44	SuccessfulCreate	Created pod: isvc-sklearn-v2-predictor-84556fcc44-2smfh
15:42:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-84556fcc44-2smfh	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1407" already present on machine
15:42:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-84556fcc44-2smfh	Started	Started container kserve-container
15:42:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-84556fcc44-2smfh	Created	Created container: kserve-container
15:42:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-7dd9b85c64-c8wt2	Unhealthy	Readiness probe failed: Get "http://10.134.0.49:8080/v2/models/isvc-sklearn-v2-runtime/ready": dial tcp 10.134.0.49:8080: connect: connection refused
15:42:33	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:42:33	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:43:24 (x8)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-84556fcc44-2smfh	Unhealthy	Readiness probe failed: dial tcp 10.134.0.50:8080: connect: connection refused
15:43:33 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:43:33 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:43:34 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:43:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	InferenceServiceReady	InferenceService [isvc-sklearn-v2] is Ready
15:43:38	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-mixed-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-mixed-predictor-56d7d747c9 from 0 to 1
15:43:38	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-mixed-predictor-56d7d747c9	SuccessfulCreate	Created pod: isvc-sklearn-v2-mixed-predictor-56d7d747c9-zlrct
15:43:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-84556fcc44-2smfh	Killing	Stopping container kserve-container
15:43:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-mixed": the object has been modified; please apply your changes to the latest version and try again
15:43:38 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-mixed": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-mixed": the object has been modified; please apply your changes to the latest version and try again
15:43:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-56d7d747c9-zlrct	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:43:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-56d7d747c9-zlrct	Created	Created container: storage-initializer
15:43:38	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-56d7d747c9-zlrct	Started	Started container storage-initializer
15:43:38	kserve-ci-e2e-test	multus	isvc-sklearn-v2-mixed-predictor-56d7d747c9-zlrct	AddedInterface	Add eth0 [10.134.0.51/23] from ovn-kubernetes
15:43:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-56d7d747c9-zlrct	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1407" already present on machine
15:43:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-56d7d747c9-zlrct	Started	Started container kserve-container
15:43:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-56d7d747c9-zlrct	Created	Created container: kserve-container
15:44:08 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:44:08 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:44:48 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-56d7d747c9-zlrct	Unhealthy	Readiness probe failed: dial tcp 10.134.0.51:8080: connect: connection refused
15:44:53 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:44:53 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:44:58	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	InferenceServiceReady	InferenceService [isvc-sklearn-v2-mixed] is Ready
15:44:58	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow": the object has been modified; please apply your changes to the latest version and try again
15:44:58 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:44:58 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow": the object has been modified; please apply your changes to the latest version and try again
15:44:58	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-predictor-88f6f6cb7 from 0 to 1
15:44:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-56d7d747c9-zlrct	Killing	Stopping container kserve-container
15:44:58	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-predictor-88f6f6cb7	SuccessfulCreate	Created pod: isvc-tensorflow-predictor-88f6f6cb7-fjkvt
15:44:59	kserve-ci-e2e-test	multus	isvc-tensorflow-predictor-88f6f6cb7-fjkvt	AddedInterface	Add eth0 [10.132.0.29/23] from ovn-kubernetes
15:44:59	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-88f6f6cb7-fjkvt	Created	Created container: storage-initializer
15:44:59	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-88f6f6cb7-fjkvt	Started	Started container storage-initializer
15:44:59	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-88f6f6cb7-fjkvt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:45:04	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-88f6f6cb7-fjkvt	Pulling	Pulling image "tensorflow/serving:2.6.2"
15:45:08	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-88f6f6cb7-fjkvt	Started	Started container kserve-container
15:45:08	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-88f6f6cb7-fjkvt	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 3.956s (3.956s including waiting). Image size: 425873876 bytes.
15:45:08	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-88f6f6cb7-fjkvt	Created	Created container: kserve-container
15:45:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	InferenceServiceReady	InferenceService [isvc-tensorflow] is Ready
15:45:20 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:45:28 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:45:28 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:45:40	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-runtime-predictor-854bdff69c from 0 to 1
15:45:40	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-runtime-predictor-854bdff69c	SuccessfulCreate	Created pod: isvc-tensorflow-runtime-predictor-854bdff69c-s2rdr
15:45:40	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-854bdff69c-s2rdr	Started	Started container storage-initializer
15:45:40	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-854bdff69c-s2rdr	Created	Created container: storage-initializer
15:45:40	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-88f6f6cb7-fjkvt	Killing	Stopping container kserve-container
15:45:40	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-854bdff69c-s2rdr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:45:40	kserve-ci-e2e-test	multus	isvc-tensorflow-runtime-predictor-854bdff69c-s2rdr	AddedInterface	Add eth0 [10.132.0.30/23] from ovn-kubernetes
15:45:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow-runtime": the object has been modified; please apply your changes to the latest version and try again
15:45:45	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-854bdff69c-s2rdr	Created	Created container: kserve-container
15:45:45	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-854bdff69c-s2rdr	Pulled	Container image "tensorflow/serving:2.6.2" already present on machine
15:45:45	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-854bdff69c-s2rdr	Started	Started container kserve-container
15:45:47 (x2)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-854bdff69c-s2rdr	Unhealthy	Readiness probe failed: dial tcp 10.132.0.30:8080: connect: connection refused
15:45:57 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:45:57	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	InferenceServiceReady	InferenceService [isvc-tensorflow-runtime] is Ready
15:46:10 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:46:10 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-88f6f6cb7-fjkvt	Unhealthy	Readiness probe failed: dial tcp 10.132.0.29:8080: connect: connection refused
15:46:10 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:46:11	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-85f9f46646-4rt76	Created	Created container: storage-initializer
15:46:11	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-85f9f46646-4rt76	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:46:11 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	UpdateFailed	Failed to update status for InferenceService "isvc-triton": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-triton": the object has been modified; please apply your changes to the latest version and try again
15:46:11	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-85f9f46646-4rt76	Started	Started container storage-initializer
15:46:11	kserve-ci-e2e-test	multus	isvc-triton-predictor-85f9f46646-4rt76	AddedInterface	Add eth0 [10.132.0.31/23] from ovn-kubernetes
15:46:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-triton": the object has been modified; please apply your changes to the latest version and try again
15:46:11	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-854bdff69c-s2rdr	Killing	Stopping container kserve-container
15:46:11	kserve-ci-e2e-test	replicaset-controller	isvc-triton-predictor-85f9f46646	SuccessfulCreate	Created pod: isvc-triton-predictor-85f9f46646-4rt76
15:46:11	kserve-ci-e2e-test	deployment-controller	isvc-triton-predictor	ScalingReplicaSet	Scaled up replica set isvc-triton-predictor-85f9f46646 from 0 to 1
15:46:15	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-85f9f46646-4rt76	Pulling	Pulling image "nvcr.io/nvidia/tritonserver:23.05-py3"
15:48:10	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-85f9f46646-4rt76	Pulled	Successfully pulled image "nvcr.io/nvidia/tritonserver:23.05-py3" in 1m54.651s (1m54.651s including waiting). Image size: 12907074623 bytes.
15:48:10	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-85f9f46646-4rt76	Created	Created container: kserve-container
15:48:10	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-85f9f46646-4rt76	Started	Started container kserve-container
15:48:11 (x2)	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-85f9f46646-4rt76	Unhealthy	Readiness probe failed: dial tcp 10.132.0.31:8080: connect: connection refused
15:48:21 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:48:21	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	InferenceServiceReady	InferenceService [isvc-triton] is Ready
15:48:26 (x9)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:48:26 (x9)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:48:33	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-predictor-6dbc9d6d47 from 0 to 1
15:48:33	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-85f9f46646-4rt76	Killing	Stopping container kserve-container
15:48:33	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-predictor-6dbc9d6d47	SuccessfulCreate	Created pod: isvc-xgboost-predictor-6dbc9d6d47-rnkmt
15:48:33	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost": the object has been modified; please apply your changes to the latest version and try again
15:48:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-6dbc9d6d47-rnkmt	Started	Started container storage-initializer
15:48:34	kserve-ci-e2e-test	multus	isvc-xgboost-predictor-6dbc9d6d47-rnkmt	AddedInterface	Add eth0 [10.132.0.32/23] from ovn-kubernetes
15:48:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-6dbc9d6d47-rnkmt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:48:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-6dbc9d6d47-rnkmt	Created	Created container: storage-initializer
15:48:38	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-6dbc9d6d47-rnkmt	Pulling	Pulling image "kserve/xgbserver:latest"
15:48:58	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-6dbc9d6d47-rnkmt	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 19.333s (19.333s including waiting). Image size: 1306229499 bytes.
15:48:58	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-6dbc9d6d47-rnkmt	Created	Created container: kserve-container
15:48:58	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-6dbc9d6d47-rnkmt	Started	Started container kserve-container
15:49:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:49:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:49:50 (x7)	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-6dbc9d6d47-rnkmt	Unhealthy	Readiness probe failed: dial tcp 10.132.0.32:8080: connect: connection refused
15:50:00 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:50:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	InferenceServiceReady	InferenceService [isvc-xgboost] is Ready
15:50:03	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-mlserver-predictor-58b7db6668 from 0 to 1
15:50:03	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-6dbc9d6d47-rnkmt	Killing	Stopping container kserve-container
15:50:03 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:50:03 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:50:03	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-mlserver-predictor-58b7db6668	SuccessfulCreate	Created pod: isvc-xgboost-v2-mlserver-predictor-58b7db6668-khmcs
15:50:03 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
15:50:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
15:50:04	kserve-ci-e2e-test	multus	isvc-xgboost-v2-mlserver-predictor-58b7db6668-khmcs	AddedInterface	Add eth0 [10.134.0.52/23] from ovn-kubernetes
15:50:04	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-58b7db6668-khmcs	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:50:04	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-58b7db6668-khmcs	Created	Created container: storage-initializer
15:50:04	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-58b7db6668-khmcs	Started	Started container storage-initializer
15:50:08	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-58b7db6668-khmcs	Started	Started container kserve-container
15:50:08	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-58b7db6668-khmcs	Created	Created container: kserve-container
15:50:08	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-58b7db6668-khmcs	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
15:50:33 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:50:33 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:50:40 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:50:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	InferenceServiceReady	InferenceService [isvc-xgboost-v2-mlserver] is Ready
15:50:44	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-c58d48f-wjp8w	Created	Created container: storage-initializer
15:50:44	kserve-ci-e2e-test	deployment-controller	xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set xgboost-v2-mlserver-predictor-c58d48f from 0 to 1
15:50:44	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-58b7db6668-khmcs	Killing	Stopping container kserve-container
15:50:44	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
15:50:44	kserve-ci-e2e-test	replicaset-controller	xgboost-v2-mlserver-predictor-c58d48f	SuccessfulCreate	Created pod: xgboost-v2-mlserver-predictor-c58d48f-wjp8w
15:50:44	kserve-ci-e2e-test	multus	xgboost-v2-mlserver-predictor-c58d48f-wjp8w	AddedInterface	Add eth0 [10.134.0.53/23] from ovn-kubernetes
15:50:44	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-c58d48f-wjp8w	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:50:44 (x2)	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
15:50:44	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-c58d48f-wjp8w	Started	Started container storage-initializer
15:50:48	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-c58d48f-wjp8w	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
15:50:48	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-c58d48f-wjp8w	Created	Created container: kserve-container
15:50:48	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-c58d48f-wjp8w	Started	Started container kserve-container
15:50:50	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-58b7db6668-khmcs	Unhealthy	Readiness probe failed: Get "http://10.134.0.52:8080/v2/models/isvc-xgboost-v2-mlserver/ready": dial tcp 10.134.0.52:8080: connect: connection refused
15:51:14 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:51:14 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:51:20	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	InferenceServiceReady	InferenceService [xgboost-v2-mlserver] is Ready
15:51:20 (x13)	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:51:24	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-runtime": the object has been modified; please apply your changes to the latest version and try again
15:51:24	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-runtime-predictor-687c7765c9 from 0 to 1
15:51:24	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-687c7765c9-sqv4p	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:51:24	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-runtime-predictor-687c7765c9	SuccessfulCreate	Created pod: isvc-xgboost-runtime-predictor-687c7765c9-sqv4p
15:51:24	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-c58d48f-wjp8w	Killing	Stopping container kserve-container
15:51:24	kserve-ci-e2e-test	multus	isvc-xgboost-runtime-predictor-687c7765c9-sqv4p	AddedInterface	Add eth0 [10.132.0.33/23] from ovn-kubernetes
15:51:24 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-runtime": the object has been modified; please apply your changes to the latest version and try again
15:51:25	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-687c7765c9-sqv4p	Created	Created container: storage-initializer
15:51:25	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-687c7765c9-sqv4p	Started	Started container storage-initializer
15:51:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-687c7765c9-sqv4p	Pulled	Container image "kserve/xgbserver:latest" already present on machine
15:51:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-687c7765c9-sqv4p	Created	Created container: kserve-container
15:51:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-687c7765c9-sqv4p	Started	Started container kserve-container
15:51:30	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-c58d48f-wjp8w	Unhealthy	Readiness probe failed: Get "http://10.134.0.53:8080/v2/models/xgboost-v2-mlserver/ready": dial tcp 10.134.0.53:8080: connect: connection refused
15:52:09 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:52:09 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:52:21 (x7)	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-687c7765c9-sqv4p	Unhealthy	Readiness probe failed: dial tcp 10.132.0.33:8080: connect: connection refused
15:52:24	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:52:24	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:52:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-runtime] is Ready
15:52:31 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:52:34	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-runtime-predictor-b5d6966c7	SuccessfulCreate	Created pod: isvc-xgboost-v2-runtime-predictor-b5d6966c7-g9rnd
15:52:34	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-runtime-predictor-b5d6966c7 from 0 to 1
15:52:34 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
15:52:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-687c7765c9-sqv4p	Killing	Stopping container kserve-container
15:52:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
15:52:35	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-b5d6966c7-g9rnd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:52:35	kserve-ci-e2e-test	multus	isvc-xgboost-v2-runtime-predictor-b5d6966c7-g9rnd	AddedInterface	Add eth0 [10.134.0.54/23] from ovn-kubernetes
15:52:35	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-b5d6966c7-g9rnd	Created	Created container: storage-initializer
15:52:35	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-b5d6966c7-g9rnd	Started	Started container storage-initializer
15:52:38	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-b5d6966c7-g9rnd	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
15:52:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-b5d6966c7-g9rnd	Created	Created container: kserve-container
15:52:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-b5d6966c7-g9rnd	Started	Started container kserve-container
15:53:04 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:53:04 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:53:21 (x2)	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-b5d6966c7-g9rnd	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
15:53:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-v2-runtime] is Ready
15:53:31 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:53:34 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:53:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-b5d6966c7-g9rnd	Killing	Stopping container kserve-container
15:53:34 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:53:34	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-predictor-5db5686f9f	SuccessfulCreate	Created pod: isvc-xgboost-v2-predictor-5db5686f9f-thlnv
15:53:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
15:53:34	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-predictor-5db5686f9f from 0 to 1
15:53:35	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5db5686f9f-thlnv	Started	Started container storage-initializer
15:53:35	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5db5686f9f-thlnv	Created	Created container: storage-initializer
15:53:35	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5db5686f9f-thlnv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:53:35	kserve-ci-e2e-test	multus	isvc-xgboost-v2-predictor-5db5686f9f-thlnv	AddedInterface	Add eth0 [10.132.0.34/23] from ovn-kubernetes
15:53:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5db5686f9f-thlnv	Pulled	Container image "kserve/xgbserver:latest" already present on machine
15:53:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5db5686f9f-thlnv	Started	Started container kserve-container
15:53:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5db5686f9f-thlnv	Created	Created container: kserve-container
15:53:40	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-b5d6966c7-g9rnd	Unhealthy	Readiness probe failed: Get "http://10.134.0.54:8080/v2/models/isvc-xgboost-v2-runtime/ready": dial tcp 10.134.0.54:8080: connect: connection refused
15:54:04 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:54:04 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:54:31 (x7)	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5db5686f9f-thlnv	Unhealthy	Readiness probe failed: dial tcp 10.132.0.34:8080: connect: connection refused
15:54:34 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:54:34 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:54:41 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:54:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	InferenceServiceReady	InferenceService [isvc-xgboost-v2] is Ready
15:54:45	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-predictor-54d746bb87	SuccessfulCreate	Created pod: isvc-sklearn-s3-predictor-54d746bb87-gk6gt
15:54:45	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-predictor-54d746bb87 from 0 to 1
15:54:45	kserve-ci-e2e-test	multus	isvc-sklearn-s3-predictor-54d746bb87-gk6gt	AddedInterface	Add eth0 [10.134.0.55/23] from ovn-kubernetes
15:54:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-54d746bb87-gk6gt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:54:45	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5db5686f9f-thlnv	Killing	Stopping container kserve-container
15:54:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-54d746bb87-gk6gt	Created	Created container: storage-initializer
15:54:45 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3": the object has been modified; please apply your changes to the latest version and try again
15:54:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-54d746bb87-gk6gt	Started	Started container storage-initializer
15:54:45	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3": the object has been modified; please apply your changes to the latest version and try again
15:54:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-54d746bb87-gk6gt	Started	Started container kserve-container
15:54:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-54d746bb87-gk6gt	Created	Created container: kserve-container
15:54:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-54d746bb87-gk6gt	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1407" already present on machine
15:55:00	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:55:00	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:55:59	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	InferenceServiceReady	InferenceService [isvc-sklearn-s3] is Ready
15:55:59 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:56:00 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:56:00 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:56:05	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-pass-predictor-7dcd76cc75 from 0 to 1
15:56:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-7dcd76cc75-hsrgp	Created	Created container: storage-initializer
15:56:05	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-pass": the object has been modified; please apply your changes to the latest version and try again
15:56:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-7dcd76cc75-hsrgp	Started	Started container storage-initializer
15:56:05	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-pass-predictor-7dcd76cc75	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-pass-predictor-7dcd76cc75-hsrgp
15:56:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-54d746bb87-gk6gt	Killing	Stopping container kserve-container
15:56:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-7dcd76cc75-hsrgp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:56:05	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-pass-predictor-7dcd76cc75-hsrgp	AddedInterface	Add eth0 [10.134.0.56/23] from ovn-kubernetes
15:56:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-7dcd76cc75-hsrgp	Started	Started container kserve-container
15:56:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-7dcd76cc75-hsrgp	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1407" already present on machine
15:56:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-7dcd76cc75-hsrgp	Created	Created container: kserve-container
15:56:09 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-54d746bb87-gk6gt	Unhealthy	Readiness probe failed: dial tcp 10.134.0.55:8080: connect: connection refused
15:56:20	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:56:20	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:57:09 (x8)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-7dcd76cc75-hsrgp	Unhealthy	Readiness probe failed: dial tcp 10.134.0.56:8080: connect: connection refused
15:57:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-global-pass] is Ready
15:57:15 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:57:20 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:57:20 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:57:25	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-7dcd76cc75-hsrgp	Killing	Stopping container kserve-container
15:57:26	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-fail": the object has been modified; please apply your changes to the latest version and try again
15:57:26	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-fail-predictor-76467cdc57 from 0 to 1
15:57:26	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-fail-predictor-76467cdc57-xcm9j	AddedInterface	Add eth0 [10.134.0.57/23] from ovn-kubernetes
15:57:26	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-fail-predictor-76467cdc57	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-fail-predictor-76467cdc57-xcm9j
15:57:31 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-76467cdc57-xcm9j	Created	Created container: storage-initializer
15:57:31 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-76467cdc57-xcm9j	Started	Started container storage-initializer
15:57:31 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-76467cdc57-xcm9j	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:57:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-76467cdc57-xcm9j	BackOff	Back-off restarting failed container storage-initializer in pod isvc-sklearn-s3-tls-global-fail-predictor-76467cdc57-xcm9j_kserve-ci-e2e-test(bdd3b94f-e6c9-4b65-badc-258007abdc25)
15:57:35 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:57:37	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-pass": the object has been modified; please apply your changes to the latest version and try again
15:57:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-65cf754f99-lp4l7	Started	Started container storage-initializer
15:57:37	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-pass-predictor-65cf754f99 from 0 to 1
15:57:37	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-pass-predictor-65cf754f99	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-pass-predictor-65cf754f99-lp4l7
15:57:37	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-pass-predictor-65cf754f99-lp4l7	AddedInterface	Add eth0 [10.134.0.58/23] from ovn-kubernetes
15:57:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-65cf754f99-lp4l7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:57:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-65cf754f99-lp4l7	Created	Created container: storage-initializer
15:57:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-65cf754f99-lp4l7	Created	Created container: kserve-container
15:57:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-65cf754f99-lp4l7	Started	Started container kserve-container
15:57:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-65cf754f99-lp4l7	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1407" already present on machine
15:58:07 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:58:07 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:58:51	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-custom-pass] is Ready
15:58:51 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:58:52 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:58:52 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:58:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-65cf754f99-lp4l7	Killing	Stopping container kserve-container
15:58:58	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-fail": the object has been modified; please apply your changes to the latest version and try again
15:58:58	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-fail-predictor-746ccb9c65	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-fail-predictor-746ccb9c65-kspkv
15:58:58	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-fail-predictor-746ccb9c65 from 0 to 1
15:58:59	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-fail-predictor-746ccb9c65-kspkv	AddedInterface	Add eth0 [10.134.0.59/23] from ovn-kubernetes
15:59:01 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-65cf754f99-lp4l7	Unhealthy	Readiness probe failed: dial tcp 10.134.0.58:8080: connect: connection refused
15:59:06 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-746ccb9c65-kspkv	Started	Started container storage-initializer
15:59:06 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-746ccb9c65-kspkv	Created	Created container: storage-initializer
15:59:06 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-746ccb9c65-kspkv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:59:06 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:59:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-746ccb9c65-kspkv	Killing	Stopping container storage-initializer
15:59:09	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-pass": the object has been modified; please apply your changes to the latest version and try again
15:59:09	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-pass-predictor-64ddbd787d	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-pass-predictor-64ddbd787d-g8vmr
15:59:09	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-pass-predictor-64ddbd787d from 0 to 1
15:59:10	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-64ddbd787d-g8vmr	Started	Started container storage-initializer
15:59:10	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-64ddbd787d-g8vmr	Created	Created container: storage-initializer
15:59:10	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-pass-predictor-64ddbd787d-g8vmr	AddedInterface	Add eth0 [10.134.0.60/23] from ovn-kubernetes
15:59:10	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-64ddbd787d-g8vmr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
15:59:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-64ddbd787d-g8vmr	Started	Started container kserve-container
15:59:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-64ddbd787d-g8vmr	Created	Created container: kserve-container
15:59:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-64ddbd787d-g8vmr	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1407" already present on machine
15:59:54 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:59:54 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
16:00:09	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:00:09	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
16:00:13 (x8)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-64ddbd787d-g8vmr	Unhealthy	Readiness probe failed: dial tcp 10.134.0.60:8080: connect: connection refused
16:00:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-serving-pass] is Ready
16:00:18 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:00:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-64ddbd787d-g8vmr	Killing	Stopping container kserve-container
16:00:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-fail": the object has been modified; please apply your changes to the latest version and try again
16:00:20	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-fail-predictor-5b99d6b556 from 0 to 1
16:00:20	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-fail-predictor-5b99d6b556	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-fail-predictor-5b99d6b556-zqc82
16:00:21	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-fail-predictor-5b99d6b556-zqc82	AddedInterface	Add eth0 [10.134.0.61/23] from ovn-kubernetes
16:00:24 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-5b99d6b556-zqc82	Created	Created container: storage-initializer
16:00:24 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-5b99d6b556-zqc82	Started	Started container storage-initializer
16:00:24 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-5b99d6b556-zqc82	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine
16:00:29 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
16:00:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-5b99d6b556-zqc82	BackOff	Back-off restarting failed container storage-initializer in pod isvc-sklearn-s3-tls-serving-fail-predictor-5b99d6b556-zqc82_kserve-ci-e2e-test(6feeac88-5812-4814-a868-6a1bf68dea1e)