Events

Time	Namespace	Component	RelatedObject	Reason	Message
15:07:17	kserve-ci-e2e-test		isvc-sklearn-runtime-predictor-6c49b9bd4f-kcz7j	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c49b9bd4f-kcz7j to ip-10-0-129-231.ec2.internal
14:52:20	kserve-ci-e2e-test		isvc-pmml-v2-kserve-predictor-75b87ff64c-8g24w	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-75b87ff64c-8g24w to ip-10-0-129-231.ec2.internal
15:06:16	kserve-ci-e2e-test		sklearn-v2-mlserver-predictor-695d5f5568-v459n	Scheduled	Successfully assigned kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-695d5f5568-v459n to ip-10-0-129-231.ec2.internal
14:49:08	kserve-ci-e2e-test		isvc-pmml-predictor-5584ffd8c9-lpm66	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-predictor-5584ffd8c9-lpm66 to ip-10-0-129-231.ec2.internal
14:50:49	kserve-ci-e2e-test		isvc-pmml-runtime-predictor-7576f6b69f-l5n4c	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-runtime-predictor-7576f6b69f-l5n4c to ip-10-0-129-231.ec2.internal
15:04:56	kserve-ci-e2e-test		isvc-sklearn-predictor-7c68dc9d59-fbsl6	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-predictor-7c68dc9d59-fbsl6 to ip-10-0-129-231.ec2.internal
14:55:28	kserve-ci-e2e-test		isvc-init-fail-d5f013-predictor-6b9db4f877-pgrkn	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-init-fail-d5f013-predictor-6b9db4f877-pgrkn to ip-10-0-129-231.ec2.internal
14:47:04	kserve-ci-e2e-test		isvc-paddle-runtime-predictor-86b49c4466-2fwr2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-runtime-predictor-86b49c4466-2fwr2 to ip-10-0-129-231.ec2.internal
14:35:46	kserve-ci-e2e-test		message-dumper-predictor-7f66cccfb6-h4p89	Scheduled	Successfully assigned kserve-ci-e2e-test/message-dumper-predictor-7f66cccfb6-h4p89 to ip-10-0-143-187.ec2.internal
15:19:30	kserve-ci-e2e-test		isvc-xgboost-v2-runtime-predictor-b5d6966c7-mqffw	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-b5d6966c7-mqffw to ip-10-0-129-231.ec2.internal
14:57:23	kserve-ci-e2e-test		isvc-predictive-xgboost-predictor-794b54b9b4-c2k6k	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-794b54b9b4-c2k6k to ip-10-0-129-231.ec2.internal
14:37:21	kserve-ci-e2e-test		isvc-lightgbm-predictor-78c8d484d6-kr52v	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-predictor-78c8d484d6-kr52v to ip-10-0-129-231.ec2.internal
15:16:49	kserve-ci-e2e-test		isvc-xgboost-v2-mlserver-predictor-58b7db6668-xqb9c	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-58b7db6668-xqb9c to ip-10-0-129-231.ec2.internal
15:18:20	kserve-ci-e2e-test		isvc-xgboost-runtime-predictor-687c7765c9-59pgt	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-687c7765c9-59pgt to ip-10-0-129-231.ec2.internal
15:15:09	kserve-ci-e2e-test		isvc-xgboost-predictor-6dbc9d6d47-qw45f	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-predictor-6dbc9d6d47-qw45f to ip-10-0-129-231.ec2.internal
15:12:48	kserve-ci-e2e-test		isvc-triton-predictor-85f9f46646-56tmz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-triton-predictor-85f9f46646-56tmz to ip-10-0-129-231.ec2.internal
15:12:16	kserve-ci-e2e-test		isvc-tensorflow-runtime-predictor-854bdff69c-xsxv6	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-854bdff69c-xsxv6 to ip-10-0-129-231.ec2.internal
15:11:24	kserve-ci-e2e-test		isvc-tensorflow-predictor-88f6f6cb7-cp9b8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-predictor-88f6f6cb7-cp9b8 to ip-10-0-129-231.ec2.internal
15:07:54	kserve-ci-e2e-test		isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8qm49	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8qm49 to ip-10-0-129-231.ec2.internal
15:08:44	kserve-ci-e2e-test		isvc-sklearn-v2-predictor-9f94dd8ff-64xbx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9f94dd8ff-64xbx to ip-10-0-129-231.ec2.internal
14:53:51	kserve-ci-e2e-test		isvc-primary-5dfb41-predictor-5fffc54484-kbnfx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-primary-5dfb41-predictor-5fffc54484-kbnfx to ip-10-0-129-231.ec2.internal
14:55:11	kserve-ci-e2e-test		isvc-secondary-5dfb41-predictor-67587b564f-t25pg	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-secondary-5dfb41-predictor-67587b564f-t25pg to ip-10-0-129-231.ec2.internal
15:10:04	kserve-ci-e2e-test		isvc-sklearn-v2-mixed-predictor-69777647bd-vszdk	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-69777647bd-vszdk to ip-10-0-129-231.ec2.internal
15:26:05	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-pass-predictor-bdf9d7997-5jhnd	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-bdf9d7997-5jhnd to ip-10-0-129-231.ec2.internal
14:45:53	kserve-ci-e2e-test		isvc-paddle-predictor-7dddcb4bd4-zhpxp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-predictor-7dddcb4bd4-zhpxp to ip-10-0-129-231.ec2.internal
14:58:53	kserve-ci-e2e-test		isvc-predictive-lightgbm-predictor-669896799c-46pvp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-669896799c-46pvp to ip-10-0-129-231.ec2.internal
15:27:26	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-fail-predictor-5b694d9b8c-x7k7k	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5b694d9b8c-x7k7k to ip-10-0-129-231.ec2.internal
15:23:01	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-pass-predictor-5bd7488f4d-hfwd8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5bd7488f4d-hfwd8 to ip-10-0-129-231.ec2.internal
14:39:02	kserve-ci-e2e-test		isvc-lightgbm-runtime-predictor-64984c7cb-mmt4w	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-64984c7cb-mmt4w to ip-10-0-129-231.ec2.internal
14:40:32	kserve-ci-e2e-test		isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xhnmz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xhnmz to ip-10-0-129-231.ec2.internal
15:24:22	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-fail-predictor-94466cc4-zzfqr	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-94466cc4-zzfqr to ip-10-0-129-231.ec2.internal
15:24:33	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-pass-predictor-f7b55c5dd-dsjg8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-f7b55c5dd-dsjg8 to ip-10-0-129-231.ec2.internal
15:25:54	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-fail-predictor-58bb75b9f8-djcbl	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-58bb75b9f8-djcbl to ip-10-0-129-231.ec2.internal
15:21:40	kserve-ci-e2e-test		isvc-sklearn-s3-predictor-674d4f474d-q8xvh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-predictor-674d4f474d-q8xvh to ip-10-0-129-231.ec2.internal
14:34:21	kserve-ci-e2e-test		isvc-sklearn-batcher-custom-predictor-9757ddd9d-hjlzl	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9757ddd9d-hjlzl to ip-10-0-129-231.ec2.internal
15:17:40	kserve-ci-e2e-test		xgboost-v2-mlserver-predictor-c58d48f-fxj75	Scheduled	Successfully assigned kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-c58d48f-fxj75 to ip-10-0-129-231.ec2.internal
15:20:30	kserve-ci-e2e-test		isvc-xgboost-v2-predictor-5db5686f9f-slpsm	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5db5686f9f-slpsm to ip-10-0-129-231.ec2.internal
15:03:24	kserve-ci-e2e-test		isvc-predictive-lightgbm-v2-predictor-65df45cf79-xknt7	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-65df45cf79-xknt7 to ip-10-0-129-231.ec2.internal
14:44:33	kserve-ci-e2e-test		isvc-sklearn-mcp-predictor-7c84c59f84-2jx57	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7c84c59f84-2jx57 to ip-10-0-129-231.ec2.internal
15:00:23	kserve-ci-e2e-test		isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-8c94s	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-8c94s to ip-10-0-129-231.ec2.internal
14:32:36	kserve-ci-e2e-test		isvc-sklearn-batcher-predictor-788dff5dc7-g7jcp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-788dff5dc7-g7jcp to ip-10-0-129-231.ec2.internal
14:48:06	kserve-ci-e2e-test		isvc-paddle-v2-kserve-predictor-7c6bd76f7b-lh7qz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7c6bd76f7b-lh7qz to ip-10-0-129-231.ec2.internal
14:55:33	kserve-ci-e2e-test		isvc-predictive-sklearn-predictor-85bccb8945-s6qfv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-85bccb8945-s6qfv to ip-10-0-129-231.ec2.internal
14:43:53	kserve-ci-e2e-test		isvc-mlflow-v2-runtime-predictor-5497fc5f68-s68px	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5497fc5f68-s68px to ip-10-0-129-231.ec2.internal
14:35:56	kserve-ci-e2e-test		isvc-logger-predictor-544f9f4cf8-rskqq	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-logger-predictor-544f9f4cf8-rskqq to ip-10-0-129-231.ec2.internal
14:43:32	kserve-ci-e2e-test		isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-5mqk5	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-5mqk5 to ip-10-0-129-231.ec2.internal
15:01:54	kserve-ci-e2e-test		isvc-predictive-xgboost-v2-predictor-67fd65d6cb-45hw4	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-67fd65d6cb-45hw4 to ip-10-0-129-231.ec2.internal
14:32:36	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-predictor-788dff5dc7	SuccessfulCreate	Created pod: isvc-sklearn-batcher-predictor-788dff5dc7-g7jcp
14:32:36	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
14:32:36	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-predictor-788dff5dc7-g7jcp	AddedInterface	Add eth0 [10.134.0.23/23] from ovn-kubernetes
14:32:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-788dff5dc7-g7jcp	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18"
14:32:36	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
14:32:36	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-predictor-788dff5dc7 from 0 to 1
14:32:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-788dff5dc7-g7jcp	Started	Started container storage-initializer
14:32:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-788dff5dc7-g7jcp	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" in 2.79s (2.79s including waiting). Image size: 300957076 bytes.
14:32:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-788dff5dc7-g7jcp	Created	Created container: storage-initializer
14:32:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-788dff5dc7-g7jcp	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1433"
14:32:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-788dff5dc7-g7jcp	Created	Created container: kserve-container
14:32:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-788dff5dc7-g7jcp	Started	Started container kserve-container
14:32:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-788dff5dc7-g7jcp	Pulling	Pulling image "quay.io/opendatahub/kserve-agent@sha256:a5c29c96f4a50b100eb461f2aad96676c97347e2c3d4bdbb37c6548004d27dcb"
14:32:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-788dff5dc7-g7jcp	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1433" in 12.914s (12.914s including waiting). Image size: 1560705952 bytes.
14:32:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-788dff5dc7-g7jcp	Started	Started container agent
14:32:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-788dff5dc7-g7jcp	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:a5c29c96f4a50b100eb461f2aad96676c97347e2c3d4bdbb37c6548004d27dcb" in 3.147s (3.147s including waiting). Image size: 237896243 bytes.
14:32:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-788dff5dc7-g7jcp	Created	Created container: agent
14:33:21 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:33:21 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:34:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:34:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:34:11 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:34:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InferenceServiceReady	InferenceService [isvc-sklearn-batcher] is Ready
14:34:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-788dff5dc7-g7jcp	Killing	Stopping container agent
14:34:21	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-custom-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-custom-predictor-9757ddd9d from 0 to 1
14:34:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-9757ddd9d-hjlzl	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:34:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-9757ddd9d-hjlzl	Created	Created container: storage-initializer
14:34:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-788dff5dc7-g7jcp	Killing	Stopping container kserve-container
14:34:21	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher-custom": the object has been modified; please apply your changes to the latest version and try again
14:34:21 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher-custom": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher-custom": the object has been modified; please apply your changes to the latest version and try again
14:34:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-9757ddd9d-hjlzl	Started	Started container storage-initializer
14:34:21	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-custom-predictor-9757ddd9d-hjlzl	AddedInterface	Add eth0 [10.134.0.24/23] from ovn-kubernetes
14:34:21	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-custom-predictor-9757ddd9d	SuccessfulCreate	Created pod: isvc-sklearn-batcher-custom-predictor-9757ddd9d-hjlzl
14:34:25	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-9757ddd9d-hjlzl	Started	Started container kserve-container
14:34:25	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-9757ddd9d-hjlzl	Created	Created container: agent
14:34:25	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-9757ddd9d-hjlzl	Started	Started container agent
14:34:25	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-9757ddd9d-hjlzl	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:a5c29c96f4a50b100eb461f2aad96676c97347e2c3d4bdbb37c6548004d27dcb" already present on machine
14:34:25	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-9757ddd9d-hjlzl	Created	Created container: kserve-container
14:34:25	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-9757ddd9d-hjlzl	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1433" already present on machine
14:34:41 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-788dff5dc7-g7jcp	Unhealthy	Readiness probe failed: dial tcp 10.134.0.23:8080: connect: connection refused
14:34:41 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-788dff5dc7-g7jcp	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
14:35:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:35:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:35:36 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:35:36 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:35:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	InferenceServiceReady	InferenceService [isvc-sklearn-batcher-custom] is Ready
14:35:38 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:35:46	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	UpdateFailed	Failed to update status for InferenceService "message-dumper": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper": the object has been modified; please apply your changes to the latest version and try again
14:35:46	kserve-ci-e2e-test	kubelet	message-dumper-predictor-7f66cccfb6-h4p89	Pulling	Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display"
14:35:46	kserve-ci-e2e-test	multus	message-dumper-predictor-7f66cccfb6-h4p89	AddedInterface	Add eth0 [10.133.0.29/23] from ovn-kubernetes
14:35:46	kserve-ci-e2e-test	deployment-controller	message-dumper-predictor	ScalingReplicaSet	Scaled up replica set message-dumper-predictor-7f66cccfb6 from 0 to 1
14:35:46	kserve-ci-e2e-test	replicaset-controller	message-dumper-predictor-7f66cccfb6	SuccessfulCreate	Created pod: message-dumper-predictor-7f66cccfb6-h4p89
14:35:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-9757ddd9d-hjlzl	Killing	Stopping container agent
14:35:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-9757ddd9d-hjlzl	Killing	Stopping container kserve-container
14:35:47	kserve-ci-e2e-test	kubelet	message-dumper-predictor-7f66cccfb6-h4p89	Pulled	Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.228s (1.228s including waiting). Image size: 14813193 bytes.
14:35:48 (x8)	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:35:48	kserve-ci-e2e-test	kubelet	message-dumper-predictor-7f66cccfb6-h4p89	Created	Created container: kserve-container
14:35:48	kserve-ci-e2e-test	kubelet	message-dumper-predictor-7f66cccfb6-h4p89	Started	Started container kserve-container
14:35:48	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	InferenceServiceReady	InferenceService [message-dumper] is Ready
14:35:56 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	UpdateFailed	Failed to update status for InferenceService "isvc-logger": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger": the object has been modified; please apply your changes to the latest version and try again
14:35:56	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger": the object has been modified; please apply your changes to the latest version and try again
14:35:56	kserve-ci-e2e-test	deployment-controller	isvc-logger-predictor	ScalingReplicaSet	Scaled up replica set isvc-logger-predictor-544f9f4cf8 from 0 to 1
14:35:56	kserve-ci-e2e-test	replicaset-controller	isvc-logger-predictor-544f9f4cf8	SuccessfulCreate	Created pod: isvc-logger-predictor-544f9f4cf8-rskqq
14:35:57	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-544f9f4cf8-rskqq	Started	Started container storage-initializer
14:35:57	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-544f9f4cf8-rskqq	Created	Created container: storage-initializer
14:35:57	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-544f9f4cf8-rskqq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:35:57	kserve-ci-e2e-test	multus	isvc-logger-predictor-544f9f4cf8-rskqq	AddedInterface	Add eth0 [10.134.0.25/23] from ovn-kubernetes
14:36:01	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-544f9f4cf8-rskqq	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1433" already present on machine
14:36:01	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-544f9f4cf8-rskqq	Created	Created container: kserve-container
14:36:01	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-544f9f4cf8-rskqq	Started	Started container agent
14:36:01	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-544f9f4cf8-rskqq	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:a5c29c96f4a50b100eb461f2aad96676c97347e2c3d4bdbb37c6548004d27dcb" already present on machine
14:36:01	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-544f9f4cf8-rskqq	Created	Created container: agent
14:36:01	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-544f9f4cf8-rskqq	Started	Started container kserve-container
14:36:08 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-9757ddd9d-hjlzl	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
14:36:08 (x12)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-9757ddd9d-hjlzl	Unhealthy	Readiness probe failed: dial tcp 10.134.0.24:5000: connect: connection refused
14:36:16 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:36:16 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:36:41 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:36:41 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:37:11 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:37:11 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:37:14 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:37:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	InferenceServiceReady	InferenceService [isvc-logger] is Ready
14:37:21	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-predictor-78c8d484d6 from 0 to 1
14:37:21	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-predictor-78c8d484d6	SuccessfulCreate	Created pod: isvc-lightgbm-predictor-78c8d484d6-kr52v
14:37:21	kserve-ci-e2e-test	kubelet	message-dumper-predictor-7f66cccfb6-h4p89	Killing	Stopping container kserve-container
14:37:21	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-544f9f4cf8-rskqq	Killing	Stopping container agent
14:37:21	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-544f9f4cf8-rskqq	Killing	Stopping container kserve-container
14:37:21	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm": the object has been modified; please apply your changes to the latest version and try again
14:37:22	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-78c8d484d6-kr52v	Started	Started container storage-initializer
14:37:22	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-78c8d484d6-kr52v	Created	Created container: storage-initializer
14:37:22	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-78c8d484d6-kr52v	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:37:22	kserve-ci-e2e-test	multus	isvc-lightgbm-predictor-78c8d484d6-kr52v	AddedInterface	Add eth0 [10.134.0.26/23] from ovn-kubernetes
14:37:26	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-78c8d484d6-kr52v	Pulling	Pulling image "kserve/lgbserver:latest"
14:37:32	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-78c8d484d6-kr52v	Pulled	Successfully pulled image "kserve/lgbserver:latest" in 6.276s (6.276s including waiting). Image size: 606297871 bytes.
14:37:32	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-78c8d484d6-kr52v	Created	Created container: kserve-container
14:37:32	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-78c8d484d6-kr52v	Started	Started container kserve-container
14:37:44 (x12)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-544f9f4cf8-rskqq	Unhealthy	Readiness probe failed: dial tcp 10.134.0.25:8080: connect: connection refused
14:37:44 (x11)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-544f9f4cf8-rskqq	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
14:38:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:38:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:38:51 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:38:51 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:38:53 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:38:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	InferenceServiceReady	InferenceService [isvc-lightgbm] is Ready
14:39:02	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-runtime-predictor-64984c7cb from 0 to 1
14:39:02	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-runtime": the object has been modified; please apply your changes to the latest version and try again
14:39:02	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-78c8d484d6-kr52v	Killing	Stopping container kserve-container
14:39:02	kserve-ci-e2e-test	multus	isvc-lightgbm-runtime-predictor-64984c7cb-mmt4w	AddedInterface	Add eth0 [10.134.0.27/23] from ovn-kubernetes
14:39:02	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-64984c7cb-mmt4w	Created	Created container: storage-initializer
14:39:02	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-runtime-predictor-64984c7cb	SuccessfulCreate	Created pod: isvc-lightgbm-runtime-predictor-64984c7cb-mmt4w
14:39:02	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-64984c7cb-mmt4w	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:39:02	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-64984c7cb-mmt4w	Started	Started container storage-initializer
14:39:03 (x10)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-78c8d484d6-kr52v	Unhealthy	Readiness probe failed: dial tcp 10.134.0.26:8080: connect: connection refused
14:39:06	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-64984c7cb-mmt4w	Started	Started container kserve-container
14:39:06	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-64984c7cb-mmt4w	Created	Created container: kserve-container
14:39:06	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-64984c7cb-mmt4w	Pulled	Container image "kserve/lgbserver:latest" already present on machine
14:39:17	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:39:17	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:40:18 (x9)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-64984c7cb-mmt4w	Unhealthy	Readiness probe failed: dial tcp 10.134.0.27:8080: connect: connection refused
14:40:28 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:40:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-runtime] is Ready
14:40:32 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:40:32 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:40:32 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
14:40:32	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
14:40:32	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-runtime-predictor-8d44c64dc	SuccessfulCreate	Created pod: isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xhnmz
14:40:32	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-64984c7cb-mmt4w	Killing	Stopping container kserve-container
14:40:32	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-runtime-predictor-8d44c64dc from 0 to 1
14:40:33	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xhnmz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:40:33	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xhnmz	Started	Started container storage-initializer
14:40:33	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xhnmz	Created	Created container: storage-initializer
14:40:33	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xhnmz	AddedInterface	Add eth0 [10.134.0.28/23] from ovn-kubernetes
14:40:36	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xhnmz	Pulling	Pulling image "docker.io/seldonio/mlserver:1.7.1"
14:42:59	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xhnmz	Created	Created container: kserve-container
14:42:59	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xhnmz	Started	Started container kserve-container
14:42:59	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xhnmz	Pulled	Successfully pulled image "docker.io/seldonio/mlserver:1.7.1" in 2m22.588s (2m22.588s including waiting). Image size: 10890461297 bytes.
14:43:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-runtime] is Ready
14:43:31 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:43:32	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8d44c64dc-xhnmz	Killing	Stopping container kserve-container
14:43:32	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
14:43:32	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-kserve-predictor-68fc4f5b49	SuccessfulCreate	Created pod: isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-5mqk5
14:43:32	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-kserve-predictor-68fc4f5b49 from 0 to 1
14:43:32 (x12)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:43:32 (x12)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:43:33	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-5mqk5	AddedInterface	Add eth0 [10.134.0.29/23] from ovn-kubernetes
14:43:33	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-5mqk5	Started	Started container storage-initializer
14:43:33	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-5mqk5	Created	Created container: storage-initializer
14:43:33	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-5mqk5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:43:37	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-5mqk5	Started	Started container kserve-container
14:43:37	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-5mqk5	Pulled	Container image "kserve/lgbserver:latest" already present on machine
14:43:37	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-5mqk5	Created	Created container: kserve-container
14:43:39 (x2)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-5mqk5	Unhealthy	Readiness probe failed: dial tcp 10.134.0.29:8080: connect: connection refused
14:43:47	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:43:47	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:43:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-kserve] is Ready
14:43:49 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:43:53	kserve-ci-e2e-test	replicaset-controller	isvc-mlflow-v2-runtime-predictor-5497fc5f68	SuccessfulCreate	Created pod: isvc-mlflow-v2-runtime-predictor-5497fc5f68-s68px
14:43:53	kserve-ci-e2e-test	multus	isvc-mlflow-v2-runtime-predictor-5497fc5f68-s68px	AddedInterface	Add eth0 [10.134.0.30/23] from ovn-kubernetes
14:43:53	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5497fc5f68-s68px	Created	Created container: storage-initializer
14:43:53	kserve-ci-e2e-test	deployment-controller	isvc-mlflow-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-mlflow-v2-runtime-predictor-5497fc5f68 from 0 to 1
14:43:53	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-68fc4f5b49-5mqk5	Killing	Stopping container kserve-container
14:43:53	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5497fc5f68-s68px	Started	Started container storage-initializer
14:43:53	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5497fc5f68-s68px	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:43:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-mlflow-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-mlflow-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
14:43:58	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5497fc5f68-s68px	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
14:43:58	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5497fc5f68-s68px	Created	Created container: kserve-container
14:43:58	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5497fc5f68-s68px	Started	Started container kserve-container
14:44:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:44:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:44:30 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:44:30	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	InferenceServiceReady	InferenceService [isvc-mlflow-v2-runtime] is Ready
14:44:33	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-mcp-predictor-7c84c59f84	SuccessfulCreate	Created pod: isvc-sklearn-mcp-predictor-7c84c59f84-2jx57
14:44:33	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5497fc5f68-s68px	Killing	Stopping container kserve-container
14:44:33	kserve-ci-e2e-test	multus	isvc-sklearn-mcp-predictor-7c84c59f84-2jx57	AddedInterface	Add eth0 [10.134.0.31/23] from ovn-kubernetes
14:44:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7c84c59f84-2jx57	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:44:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7c84c59f84-2jx57	Created	Created container: storage-initializer
14:44:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7c84c59f84-2jx57	Started	Started container storage-initializer
14:44:33	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-mcp-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-mcp-predictor-7c84c59f84 from 0 to 1
14:44:33	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-mcp": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-mcp": the object has been modified; please apply your changes to the latest version and try again
14:44:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7c84c59f84-2jx57	Created	Created container: kserve-container
14:44:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7c84c59f84-2jx57	Started	Started container kserve-container
14:44:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7c84c59f84-2jx57	Pulling	Pulling image "quay.io/opendatahub/kserve-agent:latest"
14:44:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7c84c59f84-2jx57	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1433" already present on machine
14:44:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7c84c59f84-2jx57	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent:latest" in 2.932s (2.932s including waiting). Image size: 237801512 bytes.
14:44:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7c84c59f84-2jx57	Started	Started container kserve-agent
14:44:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7c84c59f84-2jx57	Created	Created container: kserve-agent
14:45:43	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	InferenceServiceReady	InferenceService [isvc-sklearn-mcp] is Ready
14:45:43 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:45:53	kserve-ci-e2e-test	deployment-controller	isvc-paddle-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-predictor-7dddcb4bd4 from 0 to 1
14:45:53	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-7dddcb4bd4-zhpxp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:45:53	kserve-ci-e2e-test	multus	isvc-paddle-predictor-7dddcb4bd4-zhpxp	AddedInterface	Add eth0 [10.134.0.32/23] from ovn-kubernetes
14:45:53	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-7dddcb4bd4-zhpxp	Created	Created container: storage-initializer
14:45:53	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-7dddcb4bd4-zhpxp	Started	Started container storage-initializer
14:45:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7c84c59f84-2jx57	Killing	Stopping container kserve-container
14:45:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7c84c59f84-2jx57	Killing	Stopping container kserve-agent
14:45:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	UpdateFailed	Failed to update status for InferenceService "isvc-paddle": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle": the object has been modified; please apply your changes to the latest version and try again
14:45:53	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-predictor-7dddcb4bd4	SuccessfulCreate	Created pod: isvc-paddle-predictor-7dddcb4bd4-zhpxp
14:45:59	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-7dddcb4bd4-zhpxp	Pulling	Pulling image "kserve/paddleserver:latest"
14:46:10	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-7dddcb4bd4-zhpxp	Pulled	Successfully pulled image "kserve/paddleserver:latest" in 11.272s (11.272s including waiting). Image size: 1162830074 bytes.
14:46:10	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-7dddcb4bd4-zhpxp	Created	Created container: kserve-container
14:46:10	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-7dddcb4bd4-zhpxp	Started	Started container kserve-container
14:46:13 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7c84c59f84-2jx57	Unhealthy	Readiness probe failed: Get "http://10.134.0.31:8080/v1/models/isvc-sklearn-mcp": dial tcp 10.134.0.31:8080: connect: connection refused
14:46:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:46:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:46:52 (x6)	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-7dddcb4bd4-zhpxp	Unhealthy	Readiness probe failed: dial tcp 10.134.0.32:8080: connect: connection refused
14:46:53 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:46:53 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:47:02	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	InferenceServiceReady	InferenceService [isvc-paddle] is Ready
14:47:02 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:47:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-runtime": the object has been modified; please apply your changes to the latest version and try again
14:47:04	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-7dddcb4bd4-zhpxp	Killing	Stopping container kserve-container
14:47:04	kserve-ci-e2e-test	deployment-controller	isvc-paddle-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-runtime-predictor-86b49c4466 from 0 to 1
14:47:04	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-runtime-predictor-86b49c4466	SuccessfulCreate	Created pod: isvc-paddle-runtime-predictor-86b49c4466-2fwr2
14:47:05	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-86b49c4466-2fwr2	Created	Created container: storage-initializer
14:47:05	kserve-ci-e2e-test	multus	isvc-paddle-runtime-predictor-86b49c4466-2fwr2	AddedInterface	Add eth0 [10.134.0.33/23] from ovn-kubernetes
14:47:05	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-86b49c4466-2fwr2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:47:05	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-86b49c4466-2fwr2	Started	Started container storage-initializer
14:47:11	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-86b49c4466-2fwr2	Created	Created container: kserve-container
14:47:11	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-86b49c4466-2fwr2	Pulled	Container image "kserve/paddleserver:latest" already present on machine
14:47:11	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-86b49c4466-2fwr2	Started	Started container kserve-container
14:47:19	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:47:19	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:47:53 (x6)	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-86b49c4466-2fwr2	Unhealthy	Readiness probe failed: dial tcp 10.134.0.33:8080: connect: connection refused
14:48:03 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:48:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	InferenceServiceReady	InferenceService [isvc-paddle-runtime] is Ready
14:48:04 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:48:04 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:48:06	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7c6bd76f7b-lh7qz	Started	Started container storage-initializer
14:48:06	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
14:48:06	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-86b49c4466-2fwr2	Killing	Stopping container kserve-container
14:48:06	kserve-ci-e2e-test	deployment-controller	isvc-paddle-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-v2-kserve-predictor-7c6bd76f7b from 0 to 1
14:48:06	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7c6bd76f7b-lh7qz	Created	Created container: storage-initializer
14:48:06	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7c6bd76f7b-lh7qz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:48:06	kserve-ci-e2e-test	multus	isvc-paddle-v2-kserve-predictor-7c6bd76f7b-lh7qz	AddedInterface	Add eth0 [10.134.0.34/23] from ovn-kubernetes
14:48:06	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-v2-kserve-predictor-7c6bd76f7b	SuccessfulCreate	Created pod: isvc-paddle-v2-kserve-predictor-7c6bd76f7b-lh7qz
14:48:11	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7c6bd76f7b-lh7qz	Created	Created container: kserve-container
14:48:11	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7c6bd76f7b-lh7qz	Started	Started container kserve-container
14:48:11	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7c6bd76f7b-lh7qz	Pulled	Container image "kserve/paddleserver:latest" already present on machine
14:48:21	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:48:21	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:48:53 (x6)	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7c6bd76f7b-lh7qz	Unhealthy	Readiness probe failed: dial tcp 10.134.0.34:8080: connect: connection refused
14:49:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	InferenceServiceReady	InferenceService [isvc-paddle-v2-kserve] is Ready
14:49:03 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:49:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:49:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:49:08	kserve-ci-e2e-test	deployment-controller	isvc-pmml-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-predictor-5584ffd8c9 from 0 to 1
14:49:08	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-5584ffd8c9-lpm66	Started	Started container storage-initializer
14:49:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml": the object has been modified; please apply your changes to the latest version and try again
14:49:08 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	UpdateFailed	Failed to update status for InferenceService "isvc-pmml": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml": the object has been modified; please apply your changes to the latest version and try again
14:49:08	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-predictor-5584ffd8c9	SuccessfulCreate	Created pod: isvc-pmml-predictor-5584ffd8c9-lpm66
14:49:08	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7c6bd76f7b-lh7qz	Killing	Stopping container kserve-container
14:49:08	kserve-ci-e2e-test	multus	isvc-pmml-predictor-5584ffd8c9-lpm66	AddedInterface	Add eth0 [10.134.0.35/23] from ovn-kubernetes
14:49:08	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-5584ffd8c9-lpm66	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:49:08	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-5584ffd8c9-lpm66	Created	Created container: storage-initializer
14:49:12	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-5584ffd8c9-lpm66	Pulling	Pulling image "kserve/pmmlserver:latest"
14:49:19	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-5584ffd8c9-lpm66	Pulled	Successfully pulled image "kserve/pmmlserver:latest" in 6.925s (6.925s including waiting). Image size: 800927089 bytes.
14:49:19	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-5584ffd8c9-lpm66	Created	Created container: kserve-container
14:49:19	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-5584ffd8c9-lpm66	Started	Started container kserve-container
14:49:38 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:49:38 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:50:35 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-5584ffd8c9-lpm66	Unhealthy	Readiness probe failed: dial tcp 10.134.0.35:8080: connect: connection refused
14:50:38 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:50:38 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:50:45 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:50:45	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	InferenceServiceReady	InferenceService [isvc-pmml] is Ready
14:50:49	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-7576f6b69f-l5n4c	Created	Created container: storage-initializer
14:50:49	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-runtime-predictor-7576f6b69f	SuccessfulCreate	Created pod: isvc-pmml-runtime-predictor-7576f6b69f-l5n4c
14:50:49	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-7576f6b69f-l5n4c	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:50:49	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-7576f6b69f-l5n4c	Started	Started container storage-initializer
14:50:49	kserve-ci-e2e-test	multus	isvc-pmml-runtime-predictor-7576f6b69f-l5n4c	AddedInterface	Add eth0 [10.134.0.36/23] from ovn-kubernetes
14:50:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-runtime": the object has been modified; please apply your changes to the latest version and try again
14:50:49	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-5584ffd8c9-lpm66	Killing	Stopping container kserve-container
14:50:49	kserve-ci-e2e-test	deployment-controller	isvc-pmml-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-runtime-predictor-7576f6b69f from 0 to 1
14:50:53	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-7576f6b69f-l5n4c	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
14:50:54	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-7576f6b69f-l5n4c	Created	Created container: kserve-container
14:50:54	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-7576f6b69f-l5n4c	Started	Started container kserve-container
14:51:04	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:51:04	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:52:07 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-7576f6b69f-l5n4c	Unhealthy	Readiness probe failed: dial tcp 10.134.0.36:8080: connect: connection refused
14:52:17 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:52:17	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	InferenceServiceReady	InferenceService [isvc-pmml-runtime] is Ready
14:52:19 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:52:19 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:52:20	kserve-ci-e2e-test	multus	isvc-pmml-v2-kserve-predictor-75b87ff64c-8g24w	AddedInterface	Add eth0 [10.134.0.37/23] from ovn-kubernetes
14:52:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
14:52:20	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-7576f6b69f-l5n4c	Killing	Stopping container kserve-container
14:52:20	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-75b87ff64c-8g24w	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:52:20	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-75b87ff64c-8g24w	Created	Created container: storage-initializer
14:52:20	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-v2-kserve-predictor-75b87ff64c	SuccessfulCreate	Created pod: isvc-pmml-v2-kserve-predictor-75b87ff64c-8g24w
14:52:20	kserve-ci-e2e-test	deployment-controller	isvc-pmml-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-v2-kserve-predictor-75b87ff64c from 0 to 1
14:52:21	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-75b87ff64c-8g24w	Started	Started container storage-initializer
14:52:25	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-75b87ff64c-8g24w	Created	Created container: kserve-container
14:52:25	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-75b87ff64c-8g24w	Started	Started container kserve-container
14:52:25	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-75b87ff64c-8g24w	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
14:52:50 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:52:50 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:53:37 (x9)	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-75b87ff64c-8g24w	Unhealthy	Readiness probe failed: dial tcp 10.134.0.37:8080: connect: connection refused
14:53:47 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:53:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	InferenceServiceReady	InferenceService [isvc-pmml-v2-kserve] is Ready
14:53:50 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:53:50 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:53:51	kserve-ci-e2e-test	deployment-controller	isvc-primary-5dfb41-predictor	ScalingReplicaSet	Scaled up replica set isvc-primary-5dfb41-predictor-5fffc54484 from 0 to 1
14:53:51 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-5dfb41	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-5dfb41": the object has been modified; please apply your changes to the latest version and try again
14:53:51	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-75b87ff64c-8g24w	Killing	Stopping container kserve-container
14:53:51 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-5dfb41	UpdateFailed	Failed to update status for InferenceService "isvc-primary-5dfb41": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-5dfb41": the object has been modified; please apply your changes to the latest version and try again
14:53:51	kserve-ci-e2e-test	replicaset-controller	isvc-primary-5dfb41-predictor-5fffc54484	SuccessfulCreate	Created pod: isvc-primary-5dfb41-predictor-5fffc54484-kbnfx
14:53:52	kserve-ci-e2e-test	kubelet	isvc-primary-5dfb41-predictor-5fffc54484-kbnfx	Created	Created container: storage-initializer
14:53:52	kserve-ci-e2e-test	kubelet	isvc-primary-5dfb41-predictor-5fffc54484-kbnfx	Started	Started container storage-initializer
14:53:52	kserve-ci-e2e-test	kubelet	isvc-primary-5dfb41-predictor-5fffc54484-kbnfx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:53:52	kserve-ci-e2e-test	multus	isvc-primary-5dfb41-predictor-5fffc54484-kbnfx	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
14:53:56	kserve-ci-e2e-test	kubelet	isvc-primary-5dfb41-predictor-5fffc54484-kbnfx	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1433" already present on machine
14:53:56	kserve-ci-e2e-test	kubelet	isvc-primary-5dfb41-predictor-5fffc54484-kbnfx	Created	Created container: kserve-container
14:53:56	kserve-ci-e2e-test	kubelet	isvc-primary-5dfb41-predictor-5fffc54484-kbnfx	Started	Started container kserve-container
14:54:36 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-5dfb41-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:54:36 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-5dfb41-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:55:09 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-5dfb41	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:55:09	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-5dfb41	InferenceServiceReady	InferenceService [isvc-primary-5dfb41] is Ready
14:55:11	kserve-ci-e2e-test	replicaset-controller	isvc-secondary-5dfb41-predictor-67587b564f	SuccessfulCreate	Created pod: isvc-secondary-5dfb41-predictor-67587b564f-t25pg
14:55:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-5dfb41	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-5dfb41": the object has been modified; please apply your changes to the latest version and try again
14:55:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-5dfb41	UpdateFailed	Failed to update status for InferenceService "isvc-secondary-5dfb41": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-5dfb41": the object has been modified; please apply your changes to the latest version and try again
14:55:11	kserve-ci-e2e-test	deployment-controller	isvc-secondary-5dfb41-predictor	ScalingReplicaSet	Scaled up replica set isvc-secondary-5dfb41-predictor-67587b564f from 0 to 1
14:55:12	kserve-ci-e2e-test	multus	isvc-secondary-5dfb41-predictor-67587b564f-t25pg	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
14:55:16 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-5dfb41-predictor-67587b564f-t25pg	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:55:16 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-5dfb41-predictor-67587b564f-t25pg	Created	Created container: storage-initializer
14:55:16 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-5dfb41-predictor-67587b564f-t25pg	Started	Started container storage-initializer
14:55:21 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-5dfb41-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:55:21 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-5dfb41-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:55:22	kserve-ci-e2e-test	kubelet	isvc-secondary-5dfb41-predictor-67587b564f-t25pg	BackOff	Back-off restarting failed container storage-initializer in pod isvc-secondary-5dfb41-predictor-67587b564f-t25pg_kserve-ci-e2e-test(fdeafebe-9545-4f8d-88f3-98e726f00ecd)
14:55:23 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-5dfb41	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:55:26	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-5dfb41-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:55:26	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-5dfb41-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:55:27	kserve-ci-e2e-test	kubelet	isvc-primary-5dfb41-predictor-5fffc54484-kbnfx	Killing	Stopping container kserve-container
14:55:28	kserve-ci-e2e-test	replicaset-controller	isvc-init-fail-d5f013-predictor-6b9db4f877	SuccessfulCreate	Created pod: isvc-init-fail-d5f013-predictor-6b9db4f877-pgrkn
14:55:28	kserve-ci-e2e-test	deployment-controller	isvc-init-fail-d5f013-predictor	ScalingReplicaSet	Scaled up replica set isvc-init-fail-d5f013-predictor-6b9db4f877 from 0 to 1
14:55:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-d5f013	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-d5f013": the object has been modified; please apply your changes to the latest version and try again
14:55:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-d5f013	UpdateFailed	Failed to update status for InferenceService "isvc-init-fail-d5f013": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-d5f013": the object has been modified; please apply your changes to the latest version and try again
14:55:28	kserve-ci-e2e-test	multus	isvc-init-fail-d5f013-predictor-6b9db4f877-pgrkn	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
14:55:29 (x10)	kserve-ci-e2e-test	kubelet	isvc-primary-5dfb41-predictor-5fffc54484-kbnfx	Unhealthy	Readiness probe failed: dial tcp 10.134.0.38:8080: connect: connection refused
14:55:32 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-d5f013	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:55:32 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-d5f013-predictor-6b9db4f877-pgrkn	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:55:32 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-d5f013-predictor-6b9db4f877-pgrkn	Created	Created container: storage-initializer
14:55:32 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-d5f013-predictor-6b9db4f877-pgrkn	Started	Started container storage-initializer
14:55:33	kserve-ci-e2e-test	kubelet	isvc-init-fail-d5f013-predictor-6b9db4f877-pgrkn	Killing	Stopping container storage-initializer
14:55:33	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-85bccb8945-s6qfv	Started	Started container storage-initializer
14:55:33	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-85bccb8945-s6qfv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:55:33	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn": the object has been modified; please apply your changes to the latest version and try again
14:55:33	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-predictor-85bccb8945	SuccessfulCreate	Created pod: isvc-predictive-sklearn-predictor-85bccb8945-s6qfv
14:55:33	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-predictor-85bccb8945 from 0 to 1
14:55:33	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-predictor-85bccb8945-s6qfv	AddedInterface	Add eth0 [10.134.0.41/23] from ovn-kubernetes
14:55:33	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-85bccb8945-s6qfv	Created	Created container: storage-initializer
14:55:33 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn": the object has been modified; please apply your changes to the latest version and try again
14:55:37	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-85bccb8945-s6qfv	Pulling	Pulling image "kserve/predictiveserver:latest"
14:56:01	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-85bccb8945-s6qfv	Started	Started container kserve-container
14:56:01	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-85bccb8945-s6qfv	Pulled	Successfully pulled image "kserve/predictiveserver:latest" in 23.581s (23.581s including waiting). Image size: 2324221804 bytes.
14:56:01	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-85bccb8945-s6qfv	Created	Created container: kserve-container
14:56:33 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:56:33 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:57:09 (x9)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-85bccb8945-s6qfv	Unhealthy	Readiness probe failed: dial tcp 10.134.0.41:8080: connect: connection refused
14:57:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:57:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:57:19 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:57:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	InferenceServiceReady	InferenceService [isvc-predictive-sklearn] is Ready
14:57:23	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-794b54b9b4-c2k6k	Started	Started container storage-initializer
14:57:23	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-85bccb8945-s6qfv	Killing	Stopping container kserve-container
14:57:23	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-794b54b9b4-c2k6k	Created	Created container: storage-initializer
14:57:23	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-predictor-794b54b9b4	SuccessfulCreate	Created pod: isvc-predictive-xgboost-predictor-794b54b9b4-c2k6k
14:57:23	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-794b54b9b4-c2k6k	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:57:23	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-predictor-794b54b9b4-c2k6k	AddedInterface	Add eth0 [10.134.0.42/23] from ovn-kubernetes
14:57:23	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-predictor-794b54b9b4 from 0 to 1
14:57:23	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost": the object has been modified; please apply your changes to the latest version and try again
14:57:28	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-794b54b9b4-c2k6k	Created	Created container: kserve-container
14:57:28	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-794b54b9b4-c2k6k	Started	Started container kserve-container
14:57:28	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-794b54b9b4-c2k6k	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
14:58:23 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:58:23 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:58:40 (x9)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-794b54b9b4-c2k6k	Unhealthy	Readiness probe failed: dial tcp 10.134.0.42:8080: connect: connection refused
14:58:50 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:58:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	InferenceServiceReady	InferenceService [isvc-predictive-xgboost] is Ready
14:58:53 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:58:53	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-predictor-669896799c	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-predictor-669896799c-46pvp
14:58:53	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-predictor-669896799c from 0 to 1
14:58:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm": the object has been modified; please apply your changes to the latest version and try again
14:58:53 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:58:53	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-794b54b9b4-c2k6k	Killing	Stopping container kserve-container
14:58:54	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-669896799c-46pvp	Started	Started container storage-initializer
14:58:54	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-predictor-669896799c-46pvp	AddedInterface	Add eth0 [10.134.0.43/23] from ovn-kubernetes
14:58:54	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-669896799c-46pvp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
14:58:54	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-669896799c-46pvp	Created	Created container: storage-initializer
14:58:58	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-669896799c-46pvp	Created	Created container: kserve-container
14:58:58	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-669896799c-46pvp	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
14:58:58	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-669896799c-46pvp	Started	Started container kserve-container
14:59:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:59:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:00:10 (x9)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-669896799c-46pvp	Unhealthy	Readiness probe failed: dial tcp 10.134.0.43:8080: connect: connection refused
15:00:20 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:00:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm] is Ready
15:00:23	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-v2-predictor-6fd9c49f4f	SuccessfulCreate	Created pod: isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-8c94s
15:00:23	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-v2-predictor-6fd9c49f4f from 0 to 1
15:00:23 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:00:23	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
15:00:23 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:00:23	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-669896799c-46pvp	Killing	Stopping container kserve-container
15:00:24	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-8c94s	AddedInterface	Add eth0 [10.134.0.44/23] from ovn-kubernetes
15:00:24	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-8c94s	Created	Created container: storage-initializer
15:00:24	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-8c94s	Started	Started container storage-initializer
15:00:24	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-8c94s	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:00:28	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-8c94s	Started	Started container kserve-container
15:00:28	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-8c94s	Created	Created container: kserve-container
15:00:28	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-8c94s	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
15:01:08 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:01:08 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:01:40 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-8c94s	Unhealthy	Readiness probe failed: Get "http://10.134.0.44:8080/v2/models/isvc-predictive-sklearn-v2/ready": dial tcp 10.134.0.44:8080: connect: connection refused
15:01:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	InferenceServiceReady	InferenceService [isvc-predictive-sklearn-v2] is Ready
15:01:50 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:01:53	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-v2-predictor-67fd65d6cb from 0 to 1
15:01:53 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:01:53	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-6fd9c49f4f-8c94s	Killing	Stopping container kserve-container
15:01:53 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:01:54	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-v2-predictor-67fd65d6cb	SuccessfulCreate	Created pod: isvc-predictive-xgboost-v2-predictor-67fd65d6cb-45hw4
15:01:54	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-67fd65d6cb-45hw4	Started	Started container storage-initializer
15:01:54	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-v2-predictor-67fd65d6cb-45hw4	AddedInterface	Add eth0 [10.134.0.45/23] from ovn-kubernetes
15:01:54	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-67fd65d6cb-45hw4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:01:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
15:01:54 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
15:01:54	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-67fd65d6cb-45hw4	Created	Created container: storage-initializer
15:01:59	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-67fd65d6cb-45hw4	Started	Started container kserve-container
15:01:59	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-67fd65d6cb-45hw4	Created	Created container: kserve-container
15:01:59	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-67fd65d6cb-45hw4	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
15:02:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:02:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:03:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	InferenceServiceReady	InferenceService [isvc-predictive-xgboost-v2] is Ready
15:03:14 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:03:24	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-67fd65d6cb-45hw4	Killing	Stopping container kserve-container
15:03:24	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm-v2": the object has been modified; please apply your changes to the latest version and try again
15:03:24 (x6)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-67fd65d6cb-45hw4	Unhealthy	Readiness probe failed: Get "http://10.134.0.45:8080/v2/models/isvc-predictive-xgboost-v2/ready": dial tcp 10.134.0.45:8080: connect: connection refused
15:03:24	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-v2-predictor-65df45cf79	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-v2-predictor-65df45cf79-xknt7
15:03:24 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:03:24	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-v2-predictor-65df45cf79-xknt7	AddedInterface	Add eth0 [10.134.0.46/23] from ovn-kubernetes
15:03:24	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-65df45cf79-xknt7	Created	Created container: storage-initializer
15:03:24	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-v2-predictor-65df45cf79 from 0 to 1
15:03:24	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-65df45cf79-xknt7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:03:24 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:03:24	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-65df45cf79-xknt7	Started	Started container storage-initializer
15:03:29	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-65df45cf79-xknt7	Created	Created container: kserve-container
15:03:29	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-65df45cf79-xknt7	Started	Started container kserve-container
15:03:29	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-65df45cf79-xknt7	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
15:03:39	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:03:39	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:04:48	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm-v2] is Ready
15:04:48 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:04:54	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-scheduler-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-scheduler-predictor-587584d6df from 0 to 1
15:04:54 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:04:54	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-scheduler-predictor-587584d6df	SuccessfulCreate	Created pod: isvc-sklearn-scheduler-predictor-587584d6df-mstdz
15:04:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-scheduler": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
15:04:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
15:04:54 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:04:54	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-65df45cf79-xknt7	Killing	Stopping container kserve-container
15:04:55 (x7)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:04:56	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-predictor-7c68dc9d59	SuccessfulCreate	Created pod: isvc-sklearn-predictor-7c68dc9d59-fbsl6
15:04:56	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-predictor-7c68dc9d59 from 0 to 1
15:04:56	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn": the object has been modified; please apply your changes to the latest version and try again
15:04:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7c68dc9d59-fbsl6	Started	Started container storage-initializer
15:04:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7c68dc9d59-fbsl6	Created	Created container: storage-initializer
15:04:57	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7c68dc9d59-fbsl6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:04:57	kserve-ci-e2e-test	multus	isvc-sklearn-predictor-7c68dc9d59-fbsl6	AddedInterface	Add eth0 [10.134.0.47/23] from ovn-kubernetes
15:04:58 (x6)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-65df45cf79-xknt7	Unhealthy	Readiness probe failed: Get "http://10.134.0.46:8080/v2/models/isvc-predictive-lightgbm-v2/ready": dial tcp 10.134.0.46:8080: connect: connection refused
15:05:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7c68dc9d59-fbsl6	Started	Started container kserve-container
15:05:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7c68dc9d59-fbsl6	Created	Created container: kserve-container
15:05:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7c68dc9d59-fbsl6	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1433" already present on machine
15:05:41 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:05:41 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:06:10 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:06:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	InferenceServiceReady	InferenceService [isvc-sklearn] is Ready
15:06:11 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:06:11 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:06:16	kserve-ci-e2e-test	replicaset-controller	sklearn-v2-mlserver-predictor-695d5f5568	SuccessfulCreate	Created pod: sklearn-v2-mlserver-predictor-695d5f5568-v459n
15:06:16	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7c68dc9d59-fbsl6	Killing	Stopping container kserve-container
15:06:16	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "sklearn-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "sklearn-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
15:06:16	kserve-ci-e2e-test	deployment-controller	sklearn-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set sklearn-v2-mlserver-predictor-695d5f5568 from 0 to 1
15:06:17	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-695d5f5568-v459n	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:06:17	kserve-ci-e2e-test	multus	sklearn-v2-mlserver-predictor-695d5f5568-v459n	AddedInterface	Add eth0 [10.134.0.48/23] from ovn-kubernetes
15:06:17	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-695d5f5568-v459n	Created	Created container: storage-initializer
15:06:17	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-695d5f5568-v459n	Started	Started container storage-initializer
15:06:20 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-7c68dc9d59-fbsl6	Unhealthy	Readiness probe failed: dial tcp 10.134.0.47:8080: connect: connection refused
15:06:21	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-695d5f5568-v459n	Created	Created container: kserve-container
15:06:21	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-695d5f5568-v459n	Started	Started container kserve-container
15:06:21	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-695d5f5568-v459n	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
15:06:46 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:06:46 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:07:03 (x2)	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-695d5f5568-v459n	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
15:07:13	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	InferenceServiceReady	InferenceService [sklearn-v2-mlserver] is Ready
15:07:13 (x14)	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:07:16	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-runtime-predictor-6c49b9bd4f from 0 to 1
15:07:16 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:07:16 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:07:17	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-runtime": the object has been modified; please apply your changes to the latest version and try again
15:07:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-6c49b9bd4f-kcz7j	Created	Created container: storage-initializer
15:07:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-6c49b9bd4f-kcz7j	Started	Started container storage-initializer
15:07:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-6c49b9bd4f-kcz7j	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:07:17	kserve-ci-e2e-test	multus	isvc-sklearn-runtime-predictor-6c49b9bd4f-kcz7j	AddedInterface	Add eth0 [10.134.0.49/23] from ovn-kubernetes
15:07:17	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-runtime-predictor-6c49b9bd4f	SuccessfulCreate	Created pod: isvc-sklearn-runtime-predictor-6c49b9bd4f-kcz7j
15:07:17	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-695d5f5568-v459n	Killing	Stopping container kserve-container
15:07:23	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-695d5f5568-v459n	Unhealthy	Readiness probe failed: Get "http://10.134.0.48:8080/v2/models/sklearn-v2-mlserver/ready": dial tcp 10.134.0.48:8080: connect: connection refused
15:07:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-6c49b9bd4f-kcz7j	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1433" already present on machine
15:07:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-6c49b9bd4f-kcz7j	Created	Created container: kserve-container
15:07:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-6c49b9bd4f-kcz7j	Started	Started container kserve-container
15:07:31	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:07:31	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:07:36 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-6c49b9bd4f-kcz7j	Unhealthy	Readiness probe failed: dial tcp 10.134.0.49:8080: connect: connection refused
15:07:46 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:07:46	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-runtime] is Ready
15:07:47	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:07:47	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:07:54 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
15:07:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-6c49b9bd4f-kcz7j	Killing	Stopping container kserve-container
15:07:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8qm49	Started	Started container storage-initializer
15:07:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
15:07:54	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-runtime-predictor-7dd9b85c64 from 0 to 1
15:07:54	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-runtime-predictor-7dd9b85c64	SuccessfulCreate	Created pod: isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8qm49
15:07:54	kserve-ci-e2e-test	multus	isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8qm49	AddedInterface	Add eth0 [10.134.0.50/23] from ovn-kubernetes
15:07:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8qm49	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:07:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8qm49	Created	Created container: storage-initializer
15:07:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8qm49	Started	Started container kserve-container
15:07:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8qm49	Created	Created container: kserve-container
15:07:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8qm49	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
15:08:09	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:08:09	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:08:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8qm49	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
15:08:39 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:08:39 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:08:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-v2-runtime] is Ready
15:08:41 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:08:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
15:08:44	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-predictor-9f94dd8ff from 0 to 1
15:08:44	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-predictor-9f94dd8ff	SuccessfulCreate	Created pod: isvc-sklearn-v2-predictor-9f94dd8ff-64xbx
15:08:44	kserve-ci-e2e-test	multus	isvc-sklearn-v2-predictor-9f94dd8ff-64xbx	AddedInterface	Add eth0 [10.134.0.51/23] from ovn-kubernetes
15:08:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8qm49	Killing	Stopping container kserve-container
15:08:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-9f94dd8ff-64xbx	Started	Started container storage-initializer
15:08:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-9f94dd8ff-64xbx	Created	Created container: storage-initializer
15:08:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-9f94dd8ff-64xbx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:08:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-9f94dd8ff-64xbx	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1433" already present on machine
15:08:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-9f94dd8ff-64xbx	Created	Created container: kserve-container
15:08:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-9f94dd8ff-64xbx	Started	Started container kserve-container
15:08:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-7dd9b85c64-8qm49	Unhealthy	Readiness probe failed: Get "http://10.134.0.50:8080/v2/models/isvc-sklearn-v2-runtime/ready": dial tcp 10.134.0.50:8080: connect: connection refused
15:09:14 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:09:14 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:09:50 (x8)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-9f94dd8ff-64xbx	Unhealthy	Readiness probe failed: dial tcp 10.134.0.51:8080: connect: connection refused
15:09:59 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:09:59 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:10:00 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:10:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	InferenceServiceReady	InferenceService [isvc-sklearn-v2] is Ready
15:10:04	kserve-ci-e2e-test	multus	isvc-sklearn-v2-mixed-predictor-69777647bd-vszdk	AddedInterface	Add eth0 [10.134.0.52/23] from ovn-kubernetes
15:10:04	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-mixed-predictor-69777647bd	SuccessfulCreate	Created pod: isvc-sklearn-v2-mixed-predictor-69777647bd-vszdk
15:10:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-9f94dd8ff-64xbx	Killing	Stopping container kserve-container
15:10:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-69777647bd-vszdk	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:10:04	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-mixed-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-mixed-predictor-69777647bd from 0 to 1
15:10:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-mixed": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-mixed": the object has been modified; please apply your changes to the latest version and try again
15:10:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-69777647bd-vszdk	Created	Created container: storage-initializer
15:10:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-69777647bd-vszdk	Started	Started container storage-initializer
15:10:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-69777647bd-vszdk	Started	Started container kserve-container
15:10:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-69777647bd-vszdk	Created	Created container: kserve-container
15:10:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-69777647bd-vszdk	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1433" already present on machine
15:10:34 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:10:34 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:11:11 (x8)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-69777647bd-vszdk	Unhealthy	Readiness probe failed: dial tcp 10.134.0.52:8080: connect: connection refused
15:11:19 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:11:19 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:11:21 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:11:21	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	InferenceServiceReady	InferenceService [isvc-sklearn-v2-mixed] is Ready
15:11:24 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow": the object has been modified; please apply your changes to the latest version and try again
15:11:24	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow": the object has been modified; please apply your changes to the latest version and try again
15:11:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-69777647bd-vszdk	Killing	Stopping container kserve-container
15:11:24	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-predictor-88f6f6cb7 from 0 to 1
15:11:24	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-predictor-88f6f6cb7	SuccessfulCreate	Created pod: isvc-tensorflow-predictor-88f6f6cb7-cp9b8
15:11:25	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-88f6f6cb7-cp9b8	Created	Created container: storage-initializer
15:11:25	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-88f6f6cb7-cp9b8	Started	Started container storage-initializer
15:11:25	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-88f6f6cb7-cp9b8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:11:25	kserve-ci-e2e-test	multus	isvc-tensorflow-predictor-88f6f6cb7-cp9b8	AddedInterface	Add eth0 [10.134.0.53/23] from ovn-kubernetes
15:11:30	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-88f6f6cb7-cp9b8	Pulling	Pulling image "tensorflow/serving:2.6.2"
15:11:34	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-88f6f6cb7-cp9b8	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 3.784s (3.784s including waiting). Image size: 425873876 bytes.
15:11:34	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-88f6f6cb7-cp9b8	Created	Created container: kserve-container
15:11:34	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-88f6f6cb7-cp9b8	Started	Started container kserve-container
15:11:45 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-88f6f6cb7-cp9b8	Unhealthy	Readiness probe failed: dial tcp 10.134.0.53:8080: connect: connection refused
15:11:54 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:11:54 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:11:56	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	InferenceServiceReady	InferenceService [isvc-tensorflow] is Ready
15:11:56 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:12:09	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:12:09	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:12:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow-runtime": the object has been modified; please apply your changes to the latest version and try again
15:12:16	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-854bdff69c-xsxv6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:12:16	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-854bdff69c-xsxv6	Created	Created container: storage-initializer
15:12:16	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-854bdff69c-xsxv6	Started	Started container storage-initializer
15:12:16	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-runtime-predictor-854bdff69c from 0 to 1
15:12:16	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-runtime-predictor-854bdff69c	SuccessfulCreate	Created pod: isvc-tensorflow-runtime-predictor-854bdff69c-xsxv6
15:12:16	kserve-ci-e2e-test	multus	isvc-tensorflow-runtime-predictor-854bdff69c-xsxv6	AddedInterface	Add eth0 [10.134.0.54/23] from ovn-kubernetes
15:12:16	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-88f6f6cb7-cp9b8	Killing	Stopping container kserve-container
15:12:21	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-854bdff69c-xsxv6	Pulled	Container image "tensorflow/serving:2.6.2" already present on machine
15:12:21	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-854bdff69c-xsxv6	Started	Started container kserve-container
15:12:21	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-854bdff69c-xsxv6	Created	Created container: kserve-container
15:12:23 (x2)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-854bdff69c-xsxv6	Unhealthy	Readiness probe failed: dial tcp 10.134.0.54:8080: connect: connection refused
15:12:31	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:12:31	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:12:33 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:12:33	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	InferenceServiceReady	InferenceService [isvc-tensorflow-runtime] is Ready
15:12:46	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:12:46	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:12:48	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-85f9f46646-56tmz	Created	Created container: storage-initializer
15:12:48	kserve-ci-e2e-test	replicaset-controller	isvc-triton-predictor-85f9f46646	SuccessfulCreate	Created pod: isvc-triton-predictor-85f9f46646-56tmz
15:12:48	kserve-ci-e2e-test	deployment-controller	isvc-triton-predictor	ScalingReplicaSet	Scaled up replica set isvc-triton-predictor-85f9f46646 from 0 to 1
15:12:48	kserve-ci-e2e-test	multus	isvc-triton-predictor-85f9f46646-56tmz	AddedInterface	Add eth0 [10.134.0.55/23] from ovn-kubernetes
15:12:48	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-85f9f46646-56tmz	Started	Started container storage-initializer
15:12:48	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	UpdateFailed	Failed to update status for InferenceService "isvc-triton": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-triton": the object has been modified; please apply your changes to the latest version and try again
15:12:48	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-85f9f46646-56tmz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:12:48	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-854bdff69c-xsxv6	Killing	Stopping container kserve-container
15:12:53	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-85f9f46646-56tmz	Pulling	Pulling image "nvcr.io/nvidia/tritonserver:23.05-py3"
15:14:48	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-85f9f46646-56tmz	Created	Created container: kserve-container
15:14:48	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-85f9f46646-56tmz	Started	Started container kserve-container
15:14:48	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-85f9f46646-56tmz	Pulled	Successfully pulled image "nvcr.io/nvidia/tritonserver:23.05-py3" in 1m55.06s (1m55.06s including waiting). Image size: 12907074623 bytes.
15:14:49 (x2)	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-85f9f46646-56tmz	Unhealthy	Readiness probe failed: dial tcp 10.134.0.55:8080: connect: connection refused
15:14:59 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:14:59	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	InferenceServiceReady	InferenceService [isvc-triton] is Ready
15:15:03 (x9)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:15:03 (x9)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:15:09	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-predictor-6dbc9d6d47 from 0 to 1
15:15:09	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-85f9f46646-56tmz	Killing	Stopping container kserve-container
15:15:09	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-predictor-6dbc9d6d47	SuccessfulCreate	Created pod: isvc-xgboost-predictor-6dbc9d6d47-qw45f
15:15:09	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost": the object has been modified; please apply your changes to the latest version and try again
15:15:09 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost": the object has been modified; please apply your changes to the latest version and try again
15:15:10	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-6dbc9d6d47-qw45f	Started	Started container storage-initializer
15:15:10	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-6dbc9d6d47-qw45f	Created	Created container: storage-initializer
15:15:10	kserve-ci-e2e-test	multus	isvc-xgboost-predictor-6dbc9d6d47-qw45f	AddedInterface	Add eth0 [10.134.0.56/23] from ovn-kubernetes
15:15:10	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-6dbc9d6d47-qw45f	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:15:14	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-6dbc9d6d47-qw45f	Pulling	Pulling image "kserve/xgbserver:latest"
15:15:37	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-6dbc9d6d47-qw45f	Started	Started container kserve-container
15:15:37	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-6dbc9d6d47-qw45f	Created	Created container: kserve-container
15:15:37	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-6dbc9d6d47-qw45f	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 23.202s (23.202s including waiting). Image size: 1306417403 bytes.
15:16:24 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:16:24 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:16:29 (x7)	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-6dbc9d6d47-qw45f	Unhealthy	Readiness probe failed: dial tcp 10.134.0.56:8080: connect: connection refused
15:16:39	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:16:39	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:16:39 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:16:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	InferenceServiceReady	InferenceService [isvc-xgboost] is Ready
15:16:49	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-mlserver-predictor-58b7db6668	SuccessfulCreate	Created pod: isvc-xgboost-v2-mlserver-predictor-58b7db6668-xqb9c
15:16:49	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-mlserver-predictor-58b7db6668 from 0 to 1
15:16:49 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
15:16:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
15:16:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-6dbc9d6d47-qw45f	Killing	Stopping container kserve-container
15:16:50	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-58b7db6668-xqb9c	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:16:50	kserve-ci-e2e-test	multus	isvc-xgboost-v2-mlserver-predictor-58b7db6668-xqb9c	AddedInterface	Add eth0 [10.134.0.57/23] from ovn-kubernetes
15:16:50	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-58b7db6668-xqb9c	Created	Created container: storage-initializer
15:16:50	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-58b7db6668-xqb9c	Started	Started container storage-initializer
15:16:54	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-58b7db6668-xqb9c	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
15:16:54	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-58b7db6668-xqb9c	Created	Created container: kserve-container
15:16:55	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-58b7db6668-xqb9c	Started	Started container kserve-container
15:17:04	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:17:04	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:17:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-58b7db6668-xqb9c	Unhealthy	Readiness probe failed: Get "http://10.134.0.57:8080/v2/models/isvc-xgboost-v2-mlserver/ready": dial tcp 10.134.0.57:8080: connect: connection refused
15:17:34 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:17:34 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:17:37	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	InferenceServiceReady	InferenceService [isvc-xgboost-v2-mlserver] is Ready
15:17:37 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:17:39	kserve-ci-e2e-test	deployment-controller	xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set xgboost-v2-mlserver-predictor-c58d48f from 0 to 1
15:17:40	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-c58d48f-fxj75	Created	Created container: storage-initializer
15:17:40	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
15:17:40	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-58b7db6668-xqb9c	Killing	Stopping container kserve-container
15:17:40	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-c58d48f-fxj75	Started	Started container storage-initializer
15:17:40	kserve-ci-e2e-test	multus	xgboost-v2-mlserver-predictor-c58d48f-fxj75	AddedInterface	Add eth0 [10.134.0.58/23] from ovn-kubernetes
15:17:40	kserve-ci-e2e-test	replicaset-controller	xgboost-v2-mlserver-predictor-c58d48f	SuccessfulCreate	Created pod: xgboost-v2-mlserver-predictor-c58d48f-fxj75
15:17:40	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-c58d48f-fxj75	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:17:44	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-c58d48f-fxj75	Created	Created container: kserve-container
15:17:44	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-c58d48f-fxj75	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
15:17:44	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-c58d48f-fxj75	Started	Started container kserve-container
15:18:10 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:18:10 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:18:16	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	InferenceServiceReady	InferenceService [xgboost-v2-mlserver] is Ready
15:18:16 (x12)	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:18:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-runtime": the object has been modified; please apply your changes to the latest version and try again
15:18:20	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-c58d48f-fxj75	Killing	Stopping container kserve-container
15:18:20	kserve-ci-e2e-test	multus	isvc-xgboost-runtime-predictor-687c7765c9-59pgt	AddedInterface	Add eth0 [10.134.0.59/23] from ovn-kubernetes
15:18:20	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-687c7765c9-59pgt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:18:20	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-687c7765c9-59pgt	Created	Created container: storage-initializer
15:18:20	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-687c7765c9-59pgt	Started	Started container storage-initializer
15:18:20	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-runtime-predictor-687c7765c9	SuccessfulCreate	Created pod: isvc-xgboost-runtime-predictor-687c7765c9-59pgt
15:18:20	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-runtime-predictor-687c7765c9 from 0 to 1
15:18:25	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-687c7765c9-59pgt	Started	Started container kserve-container
15:18:25	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-687c7765c9-59pgt	Pulled	Container image "kserve/xgbserver:latest" already present on machine
15:18:25	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-687c7765c9-59pgt	Created	Created container: kserve-container
15:18:26	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-c58d48f-fxj75	Unhealthy	Readiness probe failed: Get "http://10.134.0.58:8080/v2/models/xgboost-v2-mlserver/ready": dial tcp 10.134.0.58:8080: connect: connection refused
15:18:50 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:18:50 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:19:17 (x7)	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-687c7765c9-59pgt	Unhealthy	Readiness probe failed: dial tcp 10.134.0.59:8080: connect: connection refused
15:19:20 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:19:20 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:19:27	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-runtime] is Ready
15:19:27 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:19:30	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-b5d6966c7-mqffw	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:19:30	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
15:19:30	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-runtime-predictor-b5d6966c7	SuccessfulCreate	Created pod: isvc-xgboost-v2-runtime-predictor-b5d6966c7-mqffw
15:19:30	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-runtime-predictor-b5d6966c7 from 0 to 1
15:19:30	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-687c7765c9-59pgt	Killing	Stopping container kserve-container
15:19:30	kserve-ci-e2e-test	multus	isvc-xgboost-v2-runtime-predictor-b5d6966c7-mqffw	AddedInterface	Add eth0 [10.134.0.60/23] from ovn-kubernetes
15:19:31	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-b5d6966c7-mqffw	Started	Started container storage-initializer
15:19:31	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-b5d6966c7-mqffw	Created	Created container: storage-initializer
15:19:35	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-b5d6966c7-mqffw	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
15:19:35	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-b5d6966c7-mqffw	Created	Created container: kserve-container
15:19:35	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-b5d6966c7-mqffw	Started	Started container kserve-container
15:20:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:20:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:20:17 (x2)	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-b5d6966c7-mqffw	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
15:20:27	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-v2-runtime] is Ready
15:20:27 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:20:30	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-predictor-5db5686f9f from 0 to 1
15:20:30 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
15:20:30	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
15:20:30 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:20:30	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-predictor-5db5686f9f	SuccessfulCreate	Created pod: isvc-xgboost-v2-predictor-5db5686f9f-slpsm
15:20:30	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-b5d6966c7-mqffw	Killing	Stopping container kserve-container
15:20:30 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:20:31	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5db5686f9f-slpsm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:20:31	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5db5686f9f-slpsm	Created	Created container: storage-initializer
15:20:31	kserve-ci-e2e-test	multus	isvc-xgboost-v2-predictor-5db5686f9f-slpsm	AddedInterface	Add eth0 [10.134.0.61/23] from ovn-kubernetes
15:20:31	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5db5686f9f-slpsm	Started	Started container storage-initializer
15:20:35	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5db5686f9f-slpsm	Pulled	Container image "kserve/xgbserver:latest" already present on machine
15:20:35	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5db5686f9f-slpsm	Created	Created container: kserve-container
15:20:35	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5db5686f9f-slpsm	Started	Started container kserve-container
15:20:37	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-b5d6966c7-mqffw	Unhealthy	Readiness probe failed: Get "http://10.134.0.60:8080/v2/models/isvc-xgboost-v2-runtime/ready": dial tcp 10.134.0.60:8080: connect: connection refused
15:21:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:21:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:21:27 (x7)	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5db5686f9f-slpsm	Unhealthy	Readiness probe failed: dial tcp 10.134.0.61:8080: connect: connection refused
15:21:30 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:21:30 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:21:37 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:21:37	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	InferenceServiceReady	InferenceService [isvc-xgboost-v2] is Ready
15:21:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3": the object has been modified; please apply your changes to the latest version and try again
15:21:40	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-predictor-674d4f474d	SuccessfulCreate	Created pod: isvc-sklearn-s3-predictor-674d4f474d-q8xvh
15:21:40	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-predictor-674d4f474d from 0 to 1
15:21:40	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-5db5686f9f-slpsm	Killing	Stopping container kserve-container
15:21:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-674d4f474d-q8xvh	Created	Created container: storage-initializer
15:21:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-674d4f474d-q8xvh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:21:41	kserve-ci-e2e-test	multus	isvc-sklearn-s3-predictor-674d4f474d-q8xvh	AddedInterface	Add eth0 [10.134.0.62/23] from ovn-kubernetes
15:21:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-674d4f474d-q8xvh	Started	Started container storage-initializer
15:21:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-674d4f474d-q8xvh	Started	Started container kserve-container
15:21:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-674d4f474d-q8xvh	Created	Created container: kserve-container
15:21:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-674d4f474d-q8xvh	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1433" already present on machine
15:21:55	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:21:55	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:22:54 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:22:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	InferenceServiceReady	InferenceService [isvc-sklearn-s3] is Ready
15:22:55 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:22:55 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:23:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-674d4f474d-q8xvh	Killing	Stopping container kserve-container
15:23:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-5bd7488f4d-hfwd8	Started	Started container storage-initializer
15:23:01	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-pass-predictor-5bd7488f4d-hfwd8	AddedInterface	Add eth0 [10.134.0.63/23] from ovn-kubernetes
15:23:01	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-pass-predictor-5bd7488f4d from 0 to 1
15:23:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-5bd7488f4d-hfwd8	Created	Created container: storage-initializer
15:23:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-5bd7488f4d-hfwd8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:23:01 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-pass": the object has been modified; please apply your changes to the latest version and try again
15:23:01	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-pass-predictor-5bd7488f4d	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-pass-predictor-5bd7488f4d-hfwd8
15:23:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-pass": the object has been modified; please apply your changes to the latest version and try again
15:23:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-5bd7488f4d-hfwd8	Created	Created container: kserve-container
15:23:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-5bd7488f4d-hfwd8	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1433" already present on machine
15:23:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-5bd7488f4d-hfwd8	Started	Started container kserve-container
15:23:04 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-674d4f474d-q8xvh	Unhealthy	Readiness probe failed: dial tcp 10.134.0.62:8080: connect: connection refused
15:23:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:23:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:24:06 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-5bd7488f4d-hfwd8	Unhealthy	Readiness probe failed: dial tcp 10.134.0.63:8080: connect: connection refused
15:24:16 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:24:16 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:24:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-global-pass] is Ready
15:24:16 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:24:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-5bd7488f4d-hfwd8	Killing	Stopping container kserve-container
15:24:22	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-fail-predictor-94466cc4	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-fail-predictor-94466cc4-zzfqr
15:24:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-fail": the object has been modified; please apply your changes to the latest version and try again
15:24:22	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-fail-predictor-94466cc4 from 0 to 1
15:24:22	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-fail-predictor-94466cc4-zzfqr	AddedInterface	Add eth0 [10.134.0.64/23] from ovn-kubernetes
15:24:28 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-94466cc4-zzfqr	Started	Started container storage-initializer
15:24:28 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-94466cc4-zzfqr	Created	Created container: storage-initializer
15:24:28 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-94466cc4-zzfqr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:24:28 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:24:32	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-94466cc4-zzfqr	Killing	Stopping container storage-initializer
15:24:33	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-pass-predictor-f7b55c5dd-dsjg8	AddedInterface	Add eth0 [10.134.0.65/23] from ovn-kubernetes
15:24:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-f7b55c5dd-dsjg8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:24:33 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-pass": the object has been modified; please apply your changes to the latest version and try again
15:24:33	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-pass": the object has been modified; please apply your changes to the latest version and try again
15:24:33	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-pass-predictor-f7b55c5dd from 0 to 1
15:24:33	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-pass-predictor-f7b55c5dd	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-pass-predictor-f7b55c5dd-dsjg8
15:24:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-f7b55c5dd-dsjg8	Created	Created container: storage-initializer
15:24:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-f7b55c5dd-dsjg8	Started	Started container storage-initializer
15:24:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-f7b55c5dd-dsjg8	Created	Created container: kserve-container
15:24:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-f7b55c5dd-dsjg8	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1433" already present on machine
15:24:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-f7b55c5dd-dsjg8	Started	Started container kserve-container
15:25:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:25:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:25:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-custom-pass] is Ready
15:25:47 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:25:48 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:25:48 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:25:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-f7b55c5dd-dsjg8	Killing	Stopping container kserve-container
15:25:54	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-fail-predictor-58bb75b9f8 from 0 to 1
15:25:54	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-fail-predictor-58bb75b9f8-djcbl	AddedInterface	Add eth0 [10.134.0.66/23] from ovn-kubernetes
15:25:54	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-fail-predictor-58bb75b9f8	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-fail-predictor-58bb75b9f8-djcbl
15:25:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-fail": the object has been modified; please apply your changes to the latest version and try again
15:25:57 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-f7b55c5dd-dsjg8	Unhealthy	Readiness probe failed: dial tcp 10.134.0.65:8080: connect: connection refused
15:25:59 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-58bb75b9f8-djcbl	Started	Started container storage-initializer
15:25:59 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-58bb75b9f8-djcbl	Created	Created container: storage-initializer
15:25:59 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-58bb75b9f8-djcbl	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:26:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-58bb75b9f8-djcbl	BackOff	Back-off restarting failed container storage-initializer in pod isvc-sklearn-s3-tls-custom-fail-predictor-58bb75b9f8-djcbl_kserve-ci-e2e-test(a059f2a2-f7db-4589-b1ed-58a50a089e7e)
15:26:03 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:26:05	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-pass-predictor-bdf9d7997 from 0 to 1
15:26:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-bdf9d7997-5jhnd	Started	Started container storage-initializer
15:26:05	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-pass": the object has been modified; please apply your changes to the latest version and try again
15:26:05	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-pass-predictor-bdf9d7997-5jhnd	AddedInterface	Add eth0 [10.134.0.67/23] from ovn-kubernetes
15:26:05	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-pass-predictor-bdf9d7997	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-pass-predictor-bdf9d7997-5jhnd
15:26:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-bdf9d7997-5jhnd	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:26:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-bdf9d7997-5jhnd	Created	Created container: storage-initializer
15:26:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-bdf9d7997-5jhnd	Started	Started container kserve-container
15:26:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-bdf9d7997-5jhnd	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1433" already present on machine
15:26:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-bdf9d7997-5jhnd	Created	Created container: kserve-container
15:26:35 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:26:35 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
15:27:19 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
15:27:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-serving-pass] is Ready
15:27:20 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:27:20 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
15:27:25	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-bdf9d7997-5jhnd	Killing	Stopping container kserve-container
15:27:26	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-fail": the object has been modified; please apply your changes to the latest version and try again
15:27:26	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-fail-predictor-5b694d9b8c from 0 to 1
15:27:26	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-fail-predictor-5b694d9b8c	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-fail-predictor-5b694d9b8c-x7k7k
15:27:26	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-fail-predictor-5b694d9b8c-x7k7k	AddedInterface	Add eth0 [10.134.0.68/23] from ovn-kubernetes
15:27:29 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-bdf9d7997-5jhnd	Unhealthy	Readiness probe failed: dial tcp 10.134.0.67:8080: connect: connection refused
15:27:30 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-5b694d9b8c-x7k7k	Started	Started container storage-initializer
15:27:30 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-5b694d9b8c-x7k7k	Created	Created container: storage-initializer
15:27:30 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-5b694d9b8c-x7k7k	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3370fbcd366f44a8710ac5ab5a02712bec0322005baa64fee46cbff50aae7c18" already present on machine
15:27:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-5b694d9b8c-x7k7k	BackOff	Back-off restarting failed container storage-initializer in pod isvc-sklearn-s3-tls-serving-fail-predictor-5b694d9b8c-x7k7k_kserve-ci-e2e-test(a343793a-5bd5-411b-8215-dc6e88ea0fe3)
15:27:34 (x15)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.