Events

Time	Namespace	Component	RelatedObject	Reason	Message
14:13:47	kserve-ci-e2e-test		isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh to ip-10-0-139-248.ec2.internal
14:47:12	kserve-ci-e2e-test		isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767 to ip-10-0-139-248.ec2.internal
14:24:16	kserve-ci-e2e-test		isvc-primary-9eb24d-predictor-86f8cd785d-wslfn	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn to ip-10-0-139-248.ec2.internal
14:16:17	kserve-ci-e2e-test		isvc-paddle-predictor-6b8b7cfb4b-7jggq	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq to ip-10-0-139-248.ec2.internal
14:25:36	kserve-ci-e2e-test		isvc-secondary-9eb24d-predictor-b57645d49-vgm5v	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v to ip-10-0-139-248.ec2.internal
14:04:35	kserve-ci-e2e-test		isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q to ip-10-0-139-248.ec2.internal
14:14:07	kserve-ci-e2e-test		isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk to ip-10-0-139-248.ec2.internal
14:30:38	kserve-ci-e2e-test		isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6 to ip-10-0-139-248.ec2.internal
14:48:53	kserve-ci-e2e-test		isvc-xgboost-runtime-predictor-779db84d9-m4wj7	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-m4wj7 to ip-10-0-130-144.ec2.internal
14:06:20	kserve-ci-e2e-test		isvc-logger-predictor-6899674c9d-l89rx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx to ip-10-0-139-248.ec2.internal
14:17:29	kserve-ci-e2e-test		isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr to ip-10-0-139-248.ec2.internal
14:07:45	kserve-ci-e2e-test		isvc-lightgbm-predictor-bdf964bd-9mdp4	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4 to ip-10-0-139-248.ec2.internal
14:33:38	kserve-ci-e2e-test		isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9 to ip-10-0-139-248.ec2.internal
14:51:13	kserve-ci-e2e-test		isvc-xgboost-v2-predictor-6fcdd6977c-jkjm4	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-jkjm4 to ip-10-0-130-144.ec2.internal
14:50:13	kserve-ci-e2e-test		isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh to ip-10-0-139-248.ec2.internal
14:06:00	kserve-ci-e2e-test		message-dumper-predictor-c7d86bcbd-f85kj	Scheduled	Successfully assigned kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-f85kj to ip-10-0-130-144.ec2.internal
14:25:57	kserve-ci-e2e-test		isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl to ip-10-0-139-248.ec2.internal
14:03:00	kserve-ci-e2e-test		isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt to ip-10-0-139-248.ec2.internal
14:10:46	kserve-ci-e2e-test		isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5 to ip-10-0-139-248.ec2.internal
14:21:13	kserve-ci-e2e-test		isvc-pmml-runtime-predictor-67bc544947-989xz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz to ip-10-0-139-248.ec2.internal
14:45:32	kserve-ci-e2e-test		isvc-xgboost-predictor-8689c4cfcc-8vsc7	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-8vsc7 to ip-10-0-130-144.ec2.internal
14:43:10	kserve-ci-e2e-test		isvc-triton-predictor-84bb65d94b-t6ffb	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-t6ffb to ip-10-0-130-144.ec2.internal
14:36:21	kserve-ci-e2e-test		sklearn-v2-mlserver-predictor-65d8664766-zbrpk	Scheduled	Successfully assigned kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk to ip-10-0-139-248.ec2.internal
14:27:47	kserve-ci-e2e-test		isvc-predictive-xgboost-predictor-7ff98fd74d-747x9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9 to ip-10-0-139-248.ec2.internal
14:14:57	kserve-ci-e2e-test		isvc-sklearn-mcp-predictor-7445486bb-g7tsz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz to ip-10-0-139-248.ec2.internal
14:41:39	kserve-ci-e2e-test		isvc-tensorflow-predictor-6756f669d7-x2hg5	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5 to ip-10-0-139-248.ec2.internal
14:37:58	kserve-ci-e2e-test		isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb to ip-10-0-139-248.ec2.internal
14:38:58	kserve-ci-e2e-test		isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj to ip-10-0-139-248.ec2.internal
14:32:08	kserve-ci-e2e-test		isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8 to ip-10-0-139-248.ec2.internal
14:18:30	kserve-ci-e2e-test		isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc to ip-10-0-139-248.ec2.internal
14:48:02	kserve-ci-e2e-test		xgboost-v2-mlserver-predictor-7799869d6f-cldbs	Scheduled	Successfully assigned kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs to ip-10-0-139-248.ec2.internal
14:42:19	kserve-ci-e2e-test		isvc-tensorflow-runtime-predictor-8699d78cf-vc58h	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vc58h to ip-10-0-130-144.ec2.internal
14:29:07	kserve-ci-e2e-test		isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k to ip-10-0-139-248.ec2.internal
14:40:19	kserve-ci-e2e-test		isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9 to ip-10-0-139-248.ec2.internal
14:09:16	kserve-ci-e2e-test		isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg to ip-10-0-139-248.ec2.internal
14:56:48	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2 to ip-10-0-139-248.ec2.internal
14:58:09	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5 to ip-10-0-139-248.ec2.internal
14:25:52	kserve-ci-e2e-test		isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn to ip-10-0-139-248.ec2.internal
14:53:53	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs to ip-10-0-139-248.ec2.internal
14:22:44	kserve-ci-e2e-test		isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg to ip-10-0-139-248.ec2.internal
14:55:15	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr to ip-10-0-139-248.ec2.internal
14:55:26	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v to ip-10-0-139-248.ec2.internal
14:56:37	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv to ip-10-0-139-248.ec2.internal
14:52:33	kserve-ci-e2e-test		isvc-sklearn-s3-predictor-6c746f6458-zshfq	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq to ip-10-0-139-248.ec2.internal
14:37:21	kserve-ci-e2e-test		isvc-sklearn-runtime-predictor-d76b964b5-jrx4t	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t to ip-10-0-139-248.ec2.internal
14:19:32	kserve-ci-e2e-test		isvc-pmml-predictor-8bb578669-hpbck	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck to ip-10-0-139-248.ec2.internal
14:35:00	kserve-ci-e2e-test		isvc-sklearn-predictor-76db96c987-bb7gr	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr to ip-10-0-139-248.ec2.internal
14:03:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt	Started	Started container storage-initializer
14:03:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
14:03:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
14:03:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt	Created	Created container: storage-initializer
14:03:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:03:00	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt	AddedInterface	Add eth0 [10.134.0.26/23] from ovn-kubernetes
14:03:00	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-predictor-6ffbc8fbd7 from 0 to 1
14:03:00	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-predictor-6ffbc8fbd7	SuccessfulCreate	Created pod: isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt
14:03:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1501"
14:03:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1501" in 12.327s (12.327s including waiting). Image size: 1560929697 bytes.
14:03:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt	Created	Created container: kserve-container
14:03:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt	Started	Started container kserve-container
14:03:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
14:03:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt	Pulling	Pulling image "quay.io/opendatahub/kserve-agent@sha256:093b5585535b7b4890c43b46d80238c68db97d853fe4f3a7840d67abf2e0a4b5"
14:03:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.156s (2.156s including waiting). Image size: 211946088 bytes.
14:03:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt	Created	Created container: kube-rbac-proxy
14:03:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt	Started	Started container kube-rbac-proxy
14:03:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt	Started	Started container agent
14:03:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt	Created	Created container: agent
14:03:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:093b5585535b7b4890c43b46d80238c68db97d853fe4f3a7840d67abf2e0a4b5" in 3.07s (3.07s including waiting). Image size: 239688438 bytes.
14:03:30 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:03:30 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:04:26	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InferenceServiceReady	InferenceService [isvc-sklearn-batcher] is Ready
14:04:26 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:04:30 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:04:30 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:04:35	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-custom-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-custom-predictor-d7c47fb6b from 0 to 1
14:04:35	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-custom-predictor-d7c47fb6b	SuccessfulCreate	Created pod: isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q
14:04:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt	Killing	Stopping container kserve-container
14:04:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q	Started	Started container storage-initializer
14:04:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt	Killing	Stopping container kube-rbac-proxy
14:04:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt	Killing	Stopping container agent
14:04:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher-custom": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher-custom": the object has been modified; please apply your changes to the latest version and try again
14:04:35	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q	AddedInterface	Add eth0 [10.134.0.27/23] from ovn-kubernetes
14:04:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:04:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q	Created	Created container: storage-initializer
14:04:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1501" already present on machine
14:04:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q	Created	Created container: kserve-container
14:04:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q	Created	Created container: kube-rbac-proxy
14:04:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q	Started	Started container kube-rbac-proxy
14:04:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:093b5585535b7b4890c43b46d80238c68db97d853fe4f3a7840d67abf2e0a4b5" already present on machine
14:04:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q	Created	Created container: agent
14:04:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q	Started	Started container agent
14:04:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:04:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q	Started	Started container kserve-container
14:04:46 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
14:04:54 (x4)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt	Unhealthy	Readiness probe failed: Get "https://10.134.0.26:8643/healthz": dial tcp 10.134.0.26:8643: connect: connection refused
14:04:56 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt	Unhealthy	Readiness probe failed: dial tcp 10.134.0.26:8080: connect: connection refused
14:05:20 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:05:20 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:05:48	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	InferenceServiceReady	InferenceService [isvc-sklearn-batcher-custom] is Ready
14:05:48 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:05:50 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:05:50 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:06:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q	Killing	Stopping container kserve-container
14:06:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q	Killing	Stopping container agent
14:06:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q	Killing	Stopping container kube-rbac-proxy
14:06:00	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	UpdateFailed	Failed to update status for InferenceService "message-dumper": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper": the object has been modified; please apply your changes to the latest version and try again
14:06:00	kserve-ci-e2e-test	replicaset-controller	message-dumper-predictor-c7d86bcbd	SuccessfulCreate	Created pod: message-dumper-predictor-c7d86bcbd-f85kj
14:06:00	kserve-ci-e2e-test	deployment-controller	message-dumper-predictor	ScalingReplicaSet	Scaled up replica set message-dumper-predictor-c7d86bcbd from 0 to 1
14:06:01	kserve-ci-e2e-test	multus	message-dumper-predictor-c7d86bcbd-f85kj	AddedInterface	Add eth0 [10.133.0.36/23] from ovn-kubernetes
14:06:01	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-f85kj	Pulling	Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display"
14:06:02	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-f85kj	Started	Started container kserve-container
14:06:02	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-f85kj	Created	Created container: kserve-container
14:06:02	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-f85kj	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
14:06:02	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-f85kj	Pulled	Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 976ms (976ms including waiting). Image size: 14813193 bytes.
14:06:04	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-f85kj	Started	Started container kube-rbac-proxy
14:06:04	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-f85kj	Created	Created container: kube-rbac-proxy
14:06:04	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-f85kj	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.197s (2.197s including waiting). Image size: 211946088 bytes.
14:06:12	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	InferenceServiceReady	InferenceService [message-dumper] is Ready
14:06:12 (x6)	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:06:15	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:06:15	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:06:18 (x4)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q	Unhealthy	Readiness probe failed: Get "https://10.134.0.27:8643/healthz": dial tcp 10.134.0.27:8643: connect: connection refused
14:06:18 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q	Unhealthy	Readiness probe failed: dial tcp 10.134.0.27:5000: connect: connection refused
14:06:18 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
14:06:20 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	UpdateFailed	Failed to update status for InferenceService "isvc-logger": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger": the object has been modified; please apply your changes to the latest version and try again
14:06:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger": the object has been modified; please apply your changes to the latest version and try again
14:06:20	kserve-ci-e2e-test	replicaset-controller	isvc-logger-predictor-6899674c9d	SuccessfulCreate	Created pod: isvc-logger-predictor-6899674c9d-l89rx
14:06:20	kserve-ci-e2e-test	deployment-controller	isvc-logger-predictor	ScalingReplicaSet	Scaled up replica set isvc-logger-predictor-6899674c9d from 0 to 1
14:06:21	kserve-ci-e2e-test	multus	isvc-logger-predictor-6899674c9d-l89rx	AddedInterface	Add eth0 [10.134.0.28/23] from ovn-kubernetes
14:06:21	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6899674c9d-l89rx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:06:21	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6899674c9d-l89rx	Started	Started container storage-initializer
14:06:21	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6899674c9d-l89rx	Created	Created container: storage-initializer
14:06:25	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6899674c9d-l89rx	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:093b5585535b7b4890c43b46d80238c68db97d853fe4f3a7840d67abf2e0a4b5" already present on machine
14:06:25	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6899674c9d-l89rx	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:06:25	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6899674c9d-l89rx	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1501" already present on machine
14:06:25	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6899674c9d-l89rx	Created	Created container: kserve-container
14:06:25	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6899674c9d-l89rx	Started	Started container agent
14:06:25	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6899674c9d-l89rx	Created	Created container: agent
14:06:25	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6899674c9d-l89rx	Started	Started container kserve-container
14:06:25	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6899674c9d-l89rx	Created	Created container: kube-rbac-proxy
14:06:25	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6899674c9d-l89rx	Started	Started container kube-rbac-proxy
14:06:30	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:06:30	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:07:05 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:07:05 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:07:33	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	InferenceServiceReady	InferenceService [isvc-logger] is Ready
14:07:33 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:07:35 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:07:35 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:07:45	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6899674c9d-l89rx	Killing	Stopping container kserve-container
14:07:45	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm": the object has been modified; please apply your changes to the latest version and try again
14:07:45	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6899674c9d-l89rx	Killing	Stopping container agent
14:07:45	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6899674c9d-l89rx	Killing	Stopping container kube-rbac-proxy
14:07:45	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-predictor-bdf964bd from 0 to 1
14:07:45	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-predictor-bdf964bd	SuccessfulCreate	Created pod: isvc-lightgbm-predictor-bdf964bd-9mdp4
14:07:46	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-9mdp4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:07:46	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-9mdp4	Created	Created container: storage-initializer
14:07:46	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-9mdp4	Started	Started container storage-initializer
14:07:46	kserve-ci-e2e-test	multus	isvc-lightgbm-predictor-bdf964bd-9mdp4	AddedInterface	Add eth0 [10.134.0.29/23] from ovn-kubernetes
14:07:46	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-f85kj	Killing	Stopping container kube-rbac-proxy
14:07:46	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-f85kj	Killing	Stopping container kserve-container
14:07:50	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-9mdp4	Pulling	Pulling image "kserve/lgbserver:latest"
14:07:57	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-9mdp4	Pulled	Successfully pulled image "kserve/lgbserver:latest" in 6.353s (6.353s including waiting). Image size: 606841615 bytes.
14:07:57	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-9mdp4	Started	Started container kube-rbac-proxy
14:07:57	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-9mdp4	Created	Created container: kube-rbac-proxy
14:07:57	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-9mdp4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:07:57	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-9mdp4	Started	Started container kserve-container
14:07:57	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-9mdp4	Created	Created container: kserve-container
14:08:03 (x4)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6899674c9d-l89rx	Unhealthy	Readiness probe failed: Get "https://10.134.0.28:8643/healthz": dial tcp 10.134.0.28:8643: connect: connection refused
14:08:03 (x11)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6899674c9d-l89rx	Unhealthy	Readiness probe failed: dial tcp 10.134.0.28:8080: connect: connection refused
14:08:03 (x10)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-6899674c9d-l89rx	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
14:08:30 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:08:30 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:09:05 (x10)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-9mdp4	Unhealthy	Readiness probe failed: dial tcp 10.134.0.29:8080: connect: connection refused
14:09:15 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:09:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	InferenceServiceReady	InferenceService [isvc-lightgbm] is Ready
14:09:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-runtime": the object has been modified; please apply your changes to the latest version and try again
14:09:16	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-9mdp4	Killing	Stopping container kube-rbac-proxy
14:09:16 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:09:16	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-9mdp4	Killing	Stopping container kserve-container
14:09:16 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:09:16	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
14:09:16	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-runtime-predictor-749c4f6d58 from 0 to 1
14:09:16	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-runtime-predictor-749c4f6d58	SuccessfulCreate	Created pod: isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg
14:09:17	kserve-ci-e2e-test	multus	isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg	AddedInterface	Add eth0 [10.134.0.30/23] from ovn-kubernetes
14:09:17	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:09:17	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg	Created	Created container: storage-initializer
14:09:17	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg	Started	Started container storage-initializer
14:09:20	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-9mdp4	Unhealthy	Readiness probe failed: Get "https://10.134.0.29:8643/healthz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
14:09:21	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg	Started	Started container kube-rbac-proxy
14:09:21	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:09:21	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg	Created	Created container: kube-rbac-proxy
14:09:21	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg	Started	Started container kserve-container
14:09:21	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg	Created	Created container: kserve-container
14:09:21	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg	Pulled	Container image "kserve/lgbserver:latest" already present on machine
14:09:46 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:09:46 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:10:39 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:10:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-runtime] is Ready
14:10:46 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
14:10:46 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:10:46	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg	Killing	Stopping container kube-rbac-proxy
14:10:46	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg	Killing	Stopping container kserve-container
14:10:46	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
14:10:46	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-runtime-predictor-8765c9667	SuccessfulCreate	Created pod: isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5
14:10:46 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:10:46	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
14:10:46	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-runtime-predictor-8765c9667 from 0 to 1
14:10:47	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:10:47	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5	AddedInterface	Add eth0 [10.134.0.31/23] from ovn-kubernetes
14:10:47	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5	Created	Created container: storage-initializer
14:10:47	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5	Started	Started container storage-initializer
14:10:49 (x10)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg	Unhealthy	Readiness probe failed: dial tcp 10.134.0.30:8080: connect: connection refused
14:10:49	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg	Unhealthy	Readiness probe failed: Get "https://10.134.0.30:8643/healthz": dial tcp 10.134.0.30:8643: connect: connection refused
14:10:51	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5	Pulling	Pulling image "docker.io/seldonio/mlserver:1.7.1"
14:13:03	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5	Started	Started container kube-rbac-proxy
14:13:03	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5	Started	Started container kserve-container
14:13:03	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5	Created	Created container: kube-rbac-proxy
14:13:03	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:13:03	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5	Created	Created container: kserve-container
14:13:03	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5	Pulled	Successfully pulled image "docker.io/seldonio/mlserver:1.7.1" in 2m11.604s (2m11.604s including waiting). Image size: 10890461297 bytes.
14:13:31 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:13:31 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:13:41 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:13:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-runtime] is Ready
14:13:46	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5	Killing	Stopping container kserve-container
14:13:46	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:13:46	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-lightgbm-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-lightgbm-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
14:13:46	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-kserve-predictor-559bf6989 from 0 to 1
14:13:46	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:13:46	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5	Killing	Stopping container kube-rbac-proxy
14:13:47	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh	Created	Created container: storage-initializer
14:13:47	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh	AddedInterface	Add eth0 [10.134.0.32/23] from ovn-kubernetes
14:13:47	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:13:47	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh	Started	Started container storage-initializer
14:13:47	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-kserve-predictor-559bf6989	SuccessfulCreate	Created pod: isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh
14:13:52	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:13:52	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh	Started	Started container kserve-container
14:13:52	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh	Created	Created container: kube-rbac-proxy
14:13:52	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh	Started	Started container kube-rbac-proxy
14:13:52	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh	Created	Created container: kserve-container
14:13:52	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh	Pulled	Container image "kserve/lgbserver:latest" already present on machine
14:13:55 (x3)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh	Unhealthy	Readiness probe failed: dial tcp 10.134.0.32:8080: connect: connection refused
14:14:00 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:14:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-kserve] is Ready
14:14:01	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:14:01	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:14:07	kserve-ci-e2e-test	multus	isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk	AddedInterface	Add eth0 [10.134.0.33/23] from ovn-kubernetes
14:14:07	kserve-ci-e2e-test	deployment-controller	isvc-mlflow-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-mlflow-v2-runtime-predictor-5fdb47d546 from 0 to 1
14:14:07	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh	Killing	Stopping container kube-rbac-proxy
14:14:07	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh	Killing	Stopping container kserve-container
14:14:07	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:14:07	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk	Created	Created container: storage-initializer
14:14:07	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk	Started	Started container storage-initializer
14:14:07	kserve-ci-e2e-test	replicaset-controller	isvc-mlflow-v2-runtime-predictor-5fdb47d546	SuccessfulCreate	Created pod: isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk
14:14:07	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-mlflow-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-mlflow-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
14:14:12	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk	Created	Created container: kserve-container
14:14:12	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk	Created	Created container: kube-rbac-proxy
14:14:12	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
14:14:12	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk	Started	Started container kserve-container
14:14:12	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:14:12	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk	Started	Started container kube-rbac-proxy
14:14:37	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:14:37	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:14:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	InferenceServiceReady	InferenceService [isvc-mlflow-v2-runtime] is Ready
14:14:50 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:14:52 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:14:52 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:14:57	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-mcp-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-mcp-predictor-7445486bb from 0 to 1
14:14:57	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-mcp": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-mcp": the object has been modified; please apply your changes to the latest version and try again
14:14:57	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk	Killing	Stopping container kserve-container
14:14:57	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk	Killing	Stopping container kube-rbac-proxy
14:14:57	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-mcp-predictor-7445486bb	SuccessfulCreate	Created pod: isvc-sklearn-mcp-predictor-7445486bb-g7tsz
14:14:58 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7445486bb-g7tsz	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
14:14:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7445486bb-g7tsz	Created	Created container: storage-initializer
14:14:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7445486bb-g7tsz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:14:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7445486bb-g7tsz	Started	Started container storage-initializer
14:14:59	kserve-ci-e2e-test	multus	isvc-sklearn-mcp-predictor-7445486bb-g7tsz	AddedInterface	Add eth0 [10.134.0.34/23] from ovn-kubernetes
14:15:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7445486bb-g7tsz	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1501" already present on machine
14:15:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7445486bb-g7tsz	Created	Created container: kserve-container
14:15:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7445486bb-g7tsz	Started	Started container kserve-container
14:15:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7445486bb-g7tsz	Pulling	Pulling image "quay.io/opendatahub/kserve-agent:latest"
14:15:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7445486bb-g7tsz	Created	Created container: kube-rbac-proxy
14:15:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7445486bb-g7tsz	Started	Started container kube-rbac-proxy
14:15:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7445486bb-g7tsz	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent:latest" in 3.054s (3.054s including waiting). Image size: 237831719 bytes.
14:15:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7445486bb-g7tsz	Created	Created container: kserve-agent
14:15:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7445486bb-g7tsz	Started	Started container kserve-agent
14:15:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7445486bb-g7tsz	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:16:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	InferenceServiceReady	InferenceService [isvc-sklearn-mcp] is Ready
14:16:14 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:16:17	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-predictor-6b8b7cfb4b	SuccessfulCreate	Created pod: isvc-paddle-predictor-6b8b7cfb4b-7jggq
14:16:17	kserve-ci-e2e-test	deployment-controller	isvc-paddle-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-predictor-6b8b7cfb4b from 0 to 1
14:16:17	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-7jggq	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-paddle-predictor-serving-cert" not found
14:16:17	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	UpdateFailed	Failed to update status for InferenceService "isvc-paddle": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle": the object has been modified; please apply your changes to the latest version and try again
14:16:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7445486bb-g7tsz	Killing	Stopping container kube-rbac-proxy
14:16:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7445486bb-g7tsz	Killing	Stopping container kserve-agent
14:16:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7445486bb-g7tsz	Killing	Stopping container kserve-container
14:16:18	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-7jggq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:16:18	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-7jggq	Created	Created container: storage-initializer
14:16:18	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-7jggq	Started	Started container storage-initializer
14:16:18	kserve-ci-e2e-test	multus	isvc-paddle-predictor-6b8b7cfb4b-7jggq	AddedInterface	Add eth0 [10.134.0.35/23] from ovn-kubernetes
14:16:23	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-7jggq	Pulling	Pulling image "kserve/paddleserver:latest"
14:16:34	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-7jggq	Pulled	Successfully pulled image "kserve/paddleserver:latest" in 10.97s (10.97s including waiting). Image size: 1162661618 bytes.
14:16:34	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-7jggq	Started	Started container kserve-container
14:16:34	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-7jggq	Started	Started container kube-rbac-proxy
14:16:34	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-7jggq	Created	Created container: kserve-container
14:16:34	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-7jggq	Created	Created container: kube-rbac-proxy
14:16:34	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-7jggq	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:16:44 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7445486bb-g7tsz	Unhealthy	Readiness probe failed: Get "http://10.134.0.34:8080/v1/models/isvc-sklearn-mcp": dial tcp 10.134.0.34:8080: connect: connection refused
14:16:44 (x6)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-7445486bb-g7tsz	Unhealthy	Readiness probe failed: Get "https://10.134.0.34:8643/healthz": dial tcp 10.134.0.34:8643: connect: connection refused
14:17:02 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:17:02 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:17:12 (x6)	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-7jggq	Unhealthy	Readiness probe failed: dial tcp 10.134.0.35:8080: connect: connection refused
14:17:17	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:17:17	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:17:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	InferenceServiceReady	InferenceService [isvc-paddle] is Ready
14:17:22 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:17:28	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-7jggq	Killing	Stopping container kube-rbac-proxy
14:17:28	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-7jggq	Killing	Stopping container kserve-container
14:17:29	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-paddle-runtime-predictor-serving-cert" not found
14:17:29	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-runtime": the object has been modified; please apply your changes to the latest version and try again
14:17:29	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-runtime-predictor-7f4d4f9dc8	SuccessfulCreate	Created pod: isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr
14:17:29	kserve-ci-e2e-test	deployment-controller	isvc-paddle-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-runtime-predictor-7f4d4f9dc8 from 0 to 1
14:17:30	kserve-ci-e2e-test	multus	isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr	AddedInterface	Add eth0 [10.134.0.36/23] from ovn-kubernetes
14:17:30	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:17:30	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr	Created	Created container: storage-initializer
14:17:30	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr	Started	Started container storage-initializer
14:17:35	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr	Created	Created container: kserve-container
14:17:35	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr	Pulled	Container image "kserve/paddleserver:latest" already present on machine
14:17:35	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr	Started	Started container kserve-container
14:17:35	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:17:35	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr	Created	Created container: kube-rbac-proxy
14:17:35	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr	Started	Started container kube-rbac-proxy
14:18:13 (x6)	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr	Unhealthy	Readiness probe failed: dial tcp 10.134.0.36:8080: connect: connection refused
14:18:14 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:18:14 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:18:23 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:18:23	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	InferenceServiceReady	InferenceService [isvc-paddle-runtime] is Ready
14:18:29	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:18:29	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:18:30	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-v2-kserve-predictor-7dbd59854	SuccessfulCreate	Created pod: isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc
14:18:30	kserve-ci-e2e-test	deployment-controller	isvc-paddle-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-v2-kserve-predictor-7dbd59854 from 0 to 1
14:18:30	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr	Killing	Stopping container kserve-container
14:18:30	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr	Killing	Stopping container kube-rbac-proxy
14:18:30	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-paddle-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-paddle-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
14:18:31	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc	Started	Started container storage-initializer
14:18:31	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc	Created	Created container: storage-initializer
14:18:31	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:18:31	kserve-ci-e2e-test	multus	isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc	AddedInterface	Add eth0 [10.134.0.37/23] from ovn-kubernetes
14:18:36	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc	Pulled	Container image "kserve/paddleserver:latest" already present on machine
14:18:36	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc	Created	Created container: kserve-container
14:18:36	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc	Started	Started container kserve-container
14:18:36	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:18:36	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc	Created	Created container: kube-rbac-proxy
14:18:36	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc	Started	Started container kube-rbac-proxy
14:18:45	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:18:45	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:19:24 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:19:24	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	InferenceServiceReady	InferenceService [isvc-paddle-v2-kserve] is Ready
14:19:30 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:19:30 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:19:32	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-hpbck	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:19:32	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-hpbck	Started	Started container storage-initializer
14:19:32	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc	Killing	Stopping container kserve-container
14:19:32	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc	Killing	Stopping container kube-rbac-proxy
14:19:32	kserve-ci-e2e-test	deployment-controller	isvc-pmml-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-predictor-8bb578669 from 0 to 1
14:19:32	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-predictor-8bb578669	SuccessfulCreate	Created pod: isvc-pmml-predictor-8bb578669-hpbck
14:19:32	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	UpdateFailed	Failed to update status for InferenceService "isvc-pmml": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml": the object has been modified; please apply your changes to the latest version and try again
14:19:32	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-hpbck	Created	Created container: storage-initializer
14:19:32	kserve-ci-e2e-test	multus	isvc-pmml-predictor-8bb578669-hpbck	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
14:19:34 (x7)	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc	Unhealthy	Readiness probe failed: dial tcp 10.134.0.37:8080: connect: connection refused
14:19:34	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc	Unhealthy	Readiness probe failed: Get "https://10.134.0.37:8643/healthz": dial tcp 10.134.0.37:8643: connect: connection refused
14:19:36	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-hpbck	Pulling	Pulling image "kserve/pmmlserver:latest"
14:19:43	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-hpbck	Started	Started container kserve-container
14:19:43	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-hpbck	Pulled	Successfully pulled image "kserve/pmmlserver:latest" in 6.955s (6.955s including waiting). Image size: 801699205 bytes.
14:19:43	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-hpbck	Created	Created container: kserve-container
14:19:43	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-hpbck	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:19:43	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-hpbck	Created	Created container: kube-rbac-proxy
14:19:43	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-hpbck	Started	Started container kube-rbac-proxy
14:20:17 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:20:17 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:21:02 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:21:02 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:21:04 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:21:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	InferenceServiceReady	InferenceService [isvc-pmml] is Ready
14:21:13	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-runtime": the object has been modified; please apply your changes to the latest version and try again
14:21:13	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-989xz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:21:13	kserve-ci-e2e-test	deployment-controller	isvc-pmml-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-runtime-predictor-67bc544947 from 0 to 1
14:21:13 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-runtime": the object has been modified; please apply your changes to the latest version and try again
14:21:13	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-hpbck	Killing	Stopping container kserve-container
14:21:13	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-runtime-predictor-67bc544947	SuccessfulCreate	Created pod: isvc-pmml-runtime-predictor-67bc544947-989xz
14:21:13	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-hpbck	Killing	Stopping container kube-rbac-proxy
14:21:13	kserve-ci-e2e-test	multus	isvc-pmml-runtime-predictor-67bc544947-989xz	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
14:21:14 (x11)	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-hpbck	Unhealthy	Readiness probe failed: dial tcp 10.134.0.38:8080: connect: connection refused
14:21:14	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-989xz	Started	Started container storage-initializer
14:21:14	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-989xz	Created	Created container: storage-initializer
14:21:17	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-989xz	Created	Created container: kserve-container
14:21:17	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-989xz	Created	Created container: kube-rbac-proxy
14:21:17	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-989xz	Started	Started container kube-rbac-proxy
14:21:17	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-989xz	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:21:17	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-989xz	Started	Started container kserve-container
14:21:17	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-989xz	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
14:21:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:21:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:22:30 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-989xz	Unhealthy	Readiness probe failed: dial tcp 10.134.0.39:8080: connect: connection refused
14:22:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	InferenceServiceReady	InferenceService [isvc-pmml-runtime] is Ready
14:22:40 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:22:43 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:22:43 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:22:44	kserve-ci-e2e-test	deployment-controller	isvc-pmml-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-v2-kserve-predictor-6578f8ffc7 from 0 to 1
14:22:44	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-v2-kserve-predictor-6578f8ffc7	SuccessfulCreate	Created pod: isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg
14:22:44	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-989xz	Killing	Stopping container kserve-container
14:22:44	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-989xz	Killing	Stopping container kube-rbac-proxy
14:22:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-v2-kserve": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-v2-kserve": the object has been modified; please apply your changes to the latest version and try again
14:22:44	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-989xz	Unhealthy	Readiness probe failed: Get "https://10.134.0.39:8643/healthz": dial tcp 10.134.0.39:8643: connect: connection refused
14:22:45	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg	Created	Created container: storage-initializer
14:22:45	kserve-ci-e2e-test	multus	isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
14:22:45	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:22:45	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg	Started	Started container storage-initializer
14:22:50	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg	Started	Started container kube-rbac-proxy
14:22:50	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg	Started	Started container kserve-container
14:22:50	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg	Created	Created container: kserve-container
14:22:50	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
14:22:50	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg	Created	Created container: kube-rbac-proxy
14:22:50	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:23:29 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:23:29 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:24:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	InferenceServiceReady	InferenceService [isvc-pmml-v2-kserve] is Ready
14:24:08 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:24:14 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:24:14 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:24:15	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg	Killing	Stopping container kserve-container
14:24:15	kserve-ci-e2e-test	deployment-controller	isvc-primary-9eb24d-predictor	ScalingReplicaSet	Scaled up replica set isvc-primary-9eb24d-predictor-86f8cd785d from 0 to 1
14:24:15	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg	Killing	Stopping container kube-rbac-proxy
14:24:15	kserve-ci-e2e-test	replicaset-controller	isvc-primary-9eb24d-predictor-86f8cd785d	SuccessfulCreate	Created pod: isvc-primary-9eb24d-predictor-86f8cd785d-wslfn
14:24:16 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-9eb24d	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-9eb24d": the object has been modified; please apply your changes to the latest version and try again
14:24:16 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-9eb24d	UpdateFailed	Failed to update status for InferenceService "isvc-primary-9eb24d": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-9eb24d": the object has been modified; please apply your changes to the latest version and try again
14:24:17	kserve-ci-e2e-test	multus	isvc-primary-9eb24d-predictor-86f8cd785d-wslfn	AddedInterface	Add eth0 [10.134.0.41/23] from ovn-kubernetes
14:24:17	kserve-ci-e2e-test	kubelet	isvc-primary-9eb24d-predictor-86f8cd785d-wslfn	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:24:17	kserve-ci-e2e-test	kubelet	isvc-primary-9eb24d-predictor-86f8cd785d-wslfn	Started	Started container storage-initializer
14:24:17	kserve-ci-e2e-test	kubelet	isvc-primary-9eb24d-predictor-86f8cd785d-wslfn	Created	Created container: storage-initializer
14:24:18 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg	Unhealthy	Readiness probe failed: dial tcp 10.134.0.40:8080: connect: connection refused
14:24:18	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg	Unhealthy	Readiness probe failed: Get "https://10.134.0.40:8643/healthz": dial tcp 10.134.0.40:8643: connect: connection refused
14:24:21	kserve-ci-e2e-test	kubelet	isvc-primary-9eb24d-predictor-86f8cd785d-wslfn	Started	Started container kube-rbac-proxy
14:24:21	kserve-ci-e2e-test	kubelet	isvc-primary-9eb24d-predictor-86f8cd785d-wslfn	Created	Created container: kube-rbac-proxy
14:24:21	kserve-ci-e2e-test	kubelet	isvc-primary-9eb24d-predictor-86f8cd785d-wslfn	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:24:21	kserve-ci-e2e-test	kubelet	isvc-primary-9eb24d-predictor-86f8cd785d-wslfn	Started	Started container kserve-container
14:24:21	kserve-ci-e2e-test	kubelet	isvc-primary-9eb24d-predictor-86f8cd785d-wslfn	Created	Created container: kserve-container
14:24:21	kserve-ci-e2e-test	kubelet	isvc-primary-9eb24d-predictor-86f8cd785d-wslfn	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1501" already present on machine
14:24:45 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-9eb24d-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:24:45 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-9eb24d-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:25:19 (x8)	kserve-ci-e2e-test	kubelet	isvc-primary-9eb24d-predictor-86f8cd785d-wslfn	Unhealthy	Readiness probe failed: dial tcp 10.134.0.41:8080: connect: connection refused
14:25:29	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-9eb24d	InferenceServiceReady	InferenceService [isvc-primary-9eb24d] is Ready
14:25:29 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-9eb24d	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:25:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-9eb24d	UpdateFailed	Failed to update status for InferenceService "isvc-secondary-9eb24d": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-9eb24d": the object has been modified; please apply your changes to the latest version and try again
14:25:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-9eb24d	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-9eb24d": the object has been modified; please apply your changes to the latest version and try again
14:25:35	kserve-ci-e2e-test	deployment-controller	isvc-secondary-9eb24d-predictor	ScalingReplicaSet	Scaled up replica set isvc-secondary-9eb24d-predictor-b57645d49 from 0 to 1
14:25:36	kserve-ci-e2e-test	kubelet	isvc-secondary-9eb24d-predictor-b57645d49-vgm5v	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-secondary-9eb24d-predictor-serving-cert" not found
14:25:36	kserve-ci-e2e-test	replicaset-controller	isvc-secondary-9eb24d-predictor-b57645d49	SuccessfulCreate	Created pod: isvc-secondary-9eb24d-predictor-b57645d49-vgm5v
14:25:37	kserve-ci-e2e-test	multus	isvc-secondary-9eb24d-predictor-b57645d49-vgm5v	AddedInterface	Add eth0 [10.134.0.42/23] from ovn-kubernetes
14:25:40 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-9eb24d-predictor-b57645d49-vgm5v	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:25:40 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-9eb24d-predictor-b57645d49-vgm5v	Started	Started container storage-initializer
14:25:40 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-9eb24d-predictor-b57645d49-vgm5v	Created	Created container: storage-initializer
14:25:44	kserve-ci-e2e-test	kubelet	isvc-secondary-9eb24d-predictor-b57645d49-vgm5v	BackOff	Back-off restarting failed container storage-initializer in pod isvc-secondary-9eb24d-predictor-b57645d49-vgm5v_kserve-ci-e2e-test(57f2551b-c26d-4c3f-95ba-c2de8f9d991e)
14:25:45 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-9eb24d-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:25:45 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-9eb24d-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:25:47 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-9eb24d	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:25:50	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-9eb24d-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:25:50	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-9eb24d-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:25:51	kserve-ci-e2e-test	kubelet	isvc-primary-9eb24d-predictor-86f8cd785d-wslfn	Killing	Stopping container kserve-container
14:25:51	kserve-ci-e2e-test	kubelet	isvc-primary-9eb24d-predictor-86f8cd785d-wslfn	Killing	Stopping container kube-rbac-proxy
14:25:52	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-d20c43	UpdateFailed	Failed to update status for InferenceService "isvc-init-fail-d20c43": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-d20c43": the object has been modified; please apply your changes to the latest version and try again
14:25:52	kserve-ci-e2e-test	deployment-controller	isvc-init-fail-d20c43-predictor	ScalingReplicaSet	Scaled up replica set isvc-init-fail-d20c43-predictor-77cb89f4f7 from 0 to 1
14:25:52	kserve-ci-e2e-test	kubelet	isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-init-fail-d20c43-predictor-serving-cert" not found
14:25:52	kserve-ci-e2e-test	replicaset-controller	isvc-init-fail-d20c43-predictor-77cb89f4f7	SuccessfulCreate	Created pod: isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn
14:25:52	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-d20c43	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-d20c43": the object has been modified; please apply your changes to the latest version and try again
14:25:53	kserve-ci-e2e-test	multus	isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn	AddedInterface	Add eth0 [10.134.0.43/23] from ovn-kubernetes
14:25:54	kserve-ci-e2e-test	kubelet	isvc-primary-9eb24d-predictor-86f8cd785d-wslfn	Unhealthy	Readiness probe failed: Get "https://10.134.0.41:8643/healthz": dial tcp 10.134.0.41:8643: connect: connection refused
14:25:56 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:25:56 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-d20c43	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:25:56 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn	Started	Started container storage-initializer
14:25:56 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn	Created	Created container: storage-initializer
14:25:57	kserve-ci-e2e-test	kubelet	isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn	Killing	Stopping container storage-initializer
14:25:57	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn": the object has been modified; please apply your changes to the latest version and try again
14:25:57	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-predictive-sklearn-predictor-serving-cert" not found
14:25:57	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-predictor-cd7c759c9 from 0 to 1
14:25:57	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-predictor-cd7c759c9	SuccessfulCreate	Created pod: isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl
14:25:58	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl	Created	Created container: storage-initializer
14:25:58	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl	Started	Started container storage-initializer
14:25:58	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl	AddedInterface	Add eth0 [10.134.0.44/23] from ovn-kubernetes
14:25:58	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:26:02	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl	Pulling	Pulling image "kserve/predictiveserver:latest"
14:26:24	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl	Pulled	Successfully pulled image "kserve/predictiveserver:latest" in 21.369s (21.369s including waiting). Image size: 2325073472 bytes.
14:26:24	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl	Created	Created container: kserve-container
14:26:24	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl	Started	Started container kserve-container
14:26:24	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:26:24	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl	Created	Created container: kube-rbac-proxy
14:26:24	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl	Started	Started container kube-rbac-proxy
14:26:42 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:26:42 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:27:31 (x9)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl	Unhealthy	Readiness probe failed: dial tcp 10.134.0.44:8080: connect: connection refused
14:27:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	InferenceServiceReady	InferenceService [isvc-predictive-sklearn] is Ready
14:27:42 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:27:42 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:27:42 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:27:47	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-predictor-7ff98fd74d	SuccessfulCreate	Created pod: isvc-predictive-xgboost-predictor-7ff98fd74d-747x9
14:27:47	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-predictor-7ff98fd74d from 0 to 1
14:27:47	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl	Killing	Stopping container kube-rbac-proxy
14:27:47	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl	Killing	Stopping container kserve-container
14:27:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost": the object has been modified; please apply your changes to the latest version and try again
14:27:47 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost": the object has been modified; please apply your changes to the latest version and try again
14:27:48	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-747x9	Started	Started container storage-initializer
14:27:48	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-predictor-7ff98fd74d-747x9	AddedInterface	Add eth0 [10.134.0.45/23] from ovn-kubernetes
14:27:48	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-747x9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:27:48	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-747x9	Created	Created container: storage-initializer
14:27:52	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-747x9	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
14:27:52	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-747x9	Started	Started container kube-rbac-proxy
14:27:52	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-747x9	Created	Created container: kserve-container
14:27:52	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-747x9	Started	Started container kserve-container
14:27:52	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-747x9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:27:52	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-747x9	Created	Created container: kube-rbac-proxy
14:28:32 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:28:32 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:29:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	InferenceServiceReady	InferenceService [isvc-predictive-xgboost] is Ready
14:29:00 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:29:02 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:29:02 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:29:07	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-747x9	Killing	Stopping container kserve-container
14:29:07	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-predictor-75cb94f9f from 0 to 1
14:29:07	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-predictor-75cb94f9f	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k
14:29:07	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-747x9	Killing	Stopping container kube-rbac-proxy
14:29:07	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm": the object has been modified; please apply your changes to the latest version and try again
14:29:08	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k	Created	Created container: storage-initializer
14:29:08	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k	AddedInterface	Add eth0 [10.134.0.46/23] from ovn-kubernetes
14:29:08	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k	Started	Started container storage-initializer
14:29:08	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:29:10	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-747x9	Unhealthy	Readiness probe failed: Get "https://10.134.0.45:8643/healthz": dial tcp 10.134.0.45:8643: connect: connection refused
14:29:10 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-747x9	Unhealthy	Readiness probe failed: dial tcp 10.134.0.45:8080: connect: connection refused
14:29:12	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k	Created	Created container: kserve-container
14:29:12	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k	Started	Started container kserve-container
14:29:12	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k	Started	Started container kube-rbac-proxy
14:29:12	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:29:12	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k	Created	Created container: kube-rbac-proxy
14:29:12	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
14:29:52 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:29:52 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:30:30 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:30:30	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm] is Ready
14:30:37 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:30:37 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:30:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
14:30:38	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-v2-predictor-b5d4f6b79	SuccessfulCreate	Created pod: isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6
14:30:38	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k	Killing	Stopping container kube-rbac-proxy
14:30:38	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k	Killing	Stopping container kserve-container
14:30:38	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-v2-predictor-b5d4f6b79 from 0 to 1
14:30:38	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
14:30:39	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:30:39	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6	AddedInterface	Add eth0 [10.134.0.47/23] from ovn-kubernetes
14:30:39	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6	Started	Started container storage-initializer
14:30:39	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6	Created	Created container: storage-initializer
14:30:40	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k	Unhealthy	Readiness probe failed: Get "https://10.134.0.46:8643/healthz": dial tcp 10.134.0.46:8643: connect: connection refused
14:30:40 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k	Unhealthy	Readiness probe failed: dial tcp 10.134.0.46:8080: connect: connection refused
14:30:43	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6	Started	Started container kserve-container
14:30:43	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
14:30:43	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6	Created	Created container: kube-rbac-proxy
14:30:43	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6	Started	Started container kube-rbac-proxy
14:30:43	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:30:43	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6	Created	Created container: kserve-container
14:31:08 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:31:08 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:32:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	InferenceServiceReady	InferenceService [isvc-predictive-sklearn-v2] is Ready
14:32:01 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:32:08 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:32:08	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-v2-predictor-6577c65fd8 from 0 to 1
14:32:08	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8	Created	Created container: storage-initializer
14:32:08	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6	Killing	Stopping container kube-rbac-proxy
14:32:08	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6	Killing	Stopping container kserve-container
14:32:08	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8	Started	Started container storage-initializer
14:32:08 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:32:08	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:32:08	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-v2-predictor-6577c65fd8	SuccessfulCreate	Created pod: isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8
14:32:08	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8	AddedInterface	Add eth0 [10.134.0.48/23] from ovn-kubernetes
14:32:08 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
14:32:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
14:32:11	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6	Unhealthy	Readiness probe failed: Get "https://10.134.0.47:8643/healthz": dial tcp 10.134.0.47:8643: connect: connection refused
14:32:11 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6	Unhealthy	Readiness probe failed: Get "http://10.134.0.47:8080/v2/models/isvc-predictive-sklearn-v2/ready": dial tcp 10.134.0.47:8080: connect: connection refused
14:32:13	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8	Started	Started container kserve-container
14:32:13	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:32:13	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8	Created	Created container: kserve-container
14:32:13	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
14:32:13	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8	Started	Started container kube-rbac-proxy
14:32:13	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8	Created	Created container: kube-rbac-proxy
14:32:53 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:32:53 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:33:31 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:33:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	InferenceServiceReady	InferenceService [isvc-predictive-xgboost-v2] is Ready
14:33:38 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:33:38	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-v2-predictor-865b4598f7	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9
14:33:38	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8	Killing	Stopping container kserve-container
14:33:38	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8	Killing	Stopping container kube-rbac-proxy
14:33:38 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:33:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-lightgbm-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-lightgbm-v2": the object has been modified; please apply your changes to the latest version and try again
14:33:38	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-v2-predictor-865b4598f7 from 0 to 1
14:33:39	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9	AddedInterface	Add eth0 [10.134.0.49/23] from ovn-kubernetes
14:33:39	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:33:39	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9	Created	Created container: storage-initializer
14:33:39	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9	Started	Started container storage-initializer
14:33:41	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8	Unhealthy	Readiness probe failed: Get "https://10.134.0.48:8643/healthz": dial tcp 10.134.0.48:8643: connect: connection refused
14:33:41 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8	Unhealthy	Readiness probe failed: Get "http://10.134.0.48:8080/v2/models/isvc-predictive-xgboost-v2/ready": dial tcp 10.134.0.48:8080: connect: connection refused
14:33:43	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9	Started	Started container kserve-container
14:33:43	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9	Created	Created container: kube-rbac-proxy
14:33:43	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9	Started	Started container kube-rbac-proxy
14:33:43	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:33:43	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9	Created	Created container: kserve-container
14:33:43	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
14:34:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:34:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:34:51	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm-v2] is Ready
14:34:51 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:34:53 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:34:53 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:34:58	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-scheduler-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-scheduler-predictor-6c68649695 from 0 to 1
14:34:58	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9	Killing	Stopping container kube-rbac-proxy
14:34:58	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9	Killing	Stopping container kserve-container
14:34:58	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-scheduler": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
14:34:58	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-scheduler-predictor-6c68649695	SuccessfulCreate	Created pod: isvc-sklearn-scheduler-predictor-6c68649695-zm2bb
14:34:58	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scheduler": the object has been modified; please apply your changes to the latest version and try again
14:34:59 (x5)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:35:00	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-predictor-76db96c987	SuccessfulCreate	Created pod: isvc-sklearn-predictor-76db96c987-bb7gr
14:35:00	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-predictor-76db96c987 from 0 to 1
14:35:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn": the object has been modified; please apply your changes to the latest version and try again
14:35:01	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9	Unhealthy	Readiness probe failed: Get "https://10.134.0.49:8643/healthz": dial tcp 10.134.0.49:8643: connect: connection refused
14:35:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-76db96c987-bb7gr	Started	Started container storage-initializer
14:35:01 (x4)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9	Unhealthy	Readiness probe failed: Get "http://10.134.0.49:8080/v2/models/isvc-predictive-lightgbm-v2/ready": dial tcp 10.134.0.49:8080: connect: connection refused
14:35:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-76db96c987-bb7gr	Created	Created container: storage-initializer
14:35:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-76db96c987-bb7gr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:35:01	kserve-ci-e2e-test	multus	isvc-sklearn-predictor-76db96c987-bb7gr	AddedInterface	Add eth0 [10.134.0.50/23] from ovn-kubernetes
14:35:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-76db96c987-bb7gr	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1501" already present on machine
14:35:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-76db96c987-bb7gr	Created	Created container: kube-rbac-proxy
14:35:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-76db96c987-bb7gr	Started	Started container kube-rbac-proxy
14:35:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-76db96c987-bb7gr	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:35:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-76db96c987-bb7gr	Created	Created container: kserve-container
14:35:05	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-76db96c987-bb7gr	Started	Started container kserve-container
14:35:30 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:35:30 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:36:13 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:36:13	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	InferenceServiceReady	InferenceService [isvc-sklearn] is Ready
14:36:15 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:36:15 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:36:20	kserve-ci-e2e-test	deployment-controller	sklearn-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set sklearn-v2-mlserver-predictor-65d8664766 from 0 to 1
14:36:20	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "sklearn-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "sklearn-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
14:36:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-76db96c987-bb7gr	Killing	Stopping container kserve-container
14:36:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-76db96c987-bb7gr	Killing	Stopping container kube-rbac-proxy
14:36:21	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-zbrpk	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:36:21	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-zbrpk	Started	Started container storage-initializer
14:36:21	kserve-ci-e2e-test	multus	sklearn-v2-mlserver-predictor-65d8664766-zbrpk	AddedInterface	Add eth0 [10.134.0.51/23] from ovn-kubernetes
14:36:21	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-zbrpk	Created	Created container: storage-initializer
14:36:21	kserve-ci-e2e-test	replicaset-controller	sklearn-v2-mlserver-predictor-65d8664766	SuccessfulCreate	Created pod: sklearn-v2-mlserver-predictor-65d8664766-zbrpk
14:36:23 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-76db96c987-bb7gr	Unhealthy	Readiness probe failed: dial tcp 10.134.0.50:8080: connect: connection refused
14:36:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-76db96c987-bb7gr	Unhealthy	Readiness probe failed: Get "https://10.134.0.50:8643/healthz": dial tcp 10.134.0.50:8643: connect: connection refused
14:36:25	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-zbrpk	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
14:36:25	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-zbrpk	Created	Created container: kserve-container
14:36:25	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-zbrpk	Started	Started container kserve-container
14:36:25	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-zbrpk	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:36:25	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-zbrpk	Created	Created container: kube-rbac-proxy
14:36:25	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-zbrpk	Started	Started container kube-rbac-proxy
14:37:02	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-zbrpk	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
14:37:06 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:37:06 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:37:13	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	InferenceServiceReady	InferenceService [sklearn-v2-mlserver] is Ready
14:37:13 (x11)	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:37:21	kserve-ci-e2e-test	multus	isvc-sklearn-runtime-predictor-d76b964b5-jrx4t	AddedInterface	Add eth0 [10.134.0.52/23] from ovn-kubernetes
14:37:21	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-runtime-predictor-d76b964b5 from 0 to 1
14:37:21	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-zbrpk	Killing	Stopping container kserve-container
14:37:21 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:37:21 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:37:21	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-runtime-predictor-d76b964b5	SuccessfulCreate	Created pod: isvc-sklearn-runtime-predictor-d76b964b5-jrx4t
14:37:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-d76b964b5-jrx4t	Started	Started container storage-initializer
14:37:21	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-zbrpk	Killing	Stopping container kube-rbac-proxy
14:37:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-d76b964b5-jrx4t	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:37:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-d76b964b5-jrx4t	Created	Created container: storage-initializer
14:37:21	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-runtime": the object has been modified; please apply your changes to the latest version and try again
14:37:22	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-zbrpk	Unhealthy	Readiness probe failed: Get "https://10.134.0.51:8643/healthz": dial tcp 10.134.0.51:8643: connect: connection refused
14:37:23	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-zbrpk	Unhealthy	Readiness probe failed: Get "http://10.134.0.51:8080/v2/models/sklearn-v2-mlserver/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
14:37:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-d76b964b5-jrx4t	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1501" already present on machine
14:37:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-d76b964b5-jrx4t	Started	Started container kserve-container
14:37:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-d76b964b5-jrx4t	Created	Created container: kserve-container
14:37:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-d76b964b5-jrx4t	Started	Started container kube-rbac-proxy
14:37:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-d76b964b5-jrx4t	Created	Created container: kube-rbac-proxy
14:37:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-d76b964b5-jrx4t	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:37:34 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-d76b964b5-jrx4t	Unhealthy	Readiness probe failed: dial tcp 10.134.0.52:8080: connect: connection refused
14:37:44 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:37:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-runtime] is Ready
14:37:51 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:37:51 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:37:58	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
14:37:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-d76b964b5-jrx4t	Killing	Stopping container kserve-container
14:37:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-d76b964b5-jrx4t	Killing	Stopping container kube-rbac-proxy
14:37:58	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-runtime-predictor-6d84c876f4	SuccessfulCreate	Created pod: isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb
14:37:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
14:37:58	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-runtime-predictor-6d84c876f4 from 0 to 1
14:37:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:37:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-d76b964b5-jrx4t	Unhealthy	Readiness probe failed: Get "https://10.134.0.52:8643/healthz": dial tcp 10.134.0.52:8643: connect: connection refused
14:37:59	kserve-ci-e2e-test	multus	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb	AddedInterface	Add eth0 [10.134.0.53/23] from ovn-kubernetes
14:37:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb	Started	Started container storage-initializer
14:37:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb	Created	Created container: storage-initializer
14:38:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
14:38:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb	Created	Created container: kserve-container
14:38:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb	Started	Started container kube-rbac-proxy
14:38:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb	Created	Created container: kube-rbac-proxy
14:38:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:38:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb	Started	Started container kserve-container
14:38:28 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:38:28 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:38:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
14:38:51	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-v2-runtime] is Ready
14:38:51 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:38:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-v2-predictor-serving-cert" not found
14:38:58	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-predictor-5b5d98bd7f	SuccessfulCreate	Created pod: isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj
14:38:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb	Killing	Stopping container kube-rbac-proxy
14:38:58 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:38:58 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:38:58	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2": the object has been modified; please apply your changes to the latest version and try again
14:38:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb	Killing	Stopping container kserve-container
14:38:58	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-predictor-5b5d98bd7f from 0 to 1
14:38:59	kserve-ci-e2e-test	multus	isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj	AddedInterface	Add eth0 [10.134.0.54/23] from ovn-kubernetes
14:38:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:38:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj	Created	Created container: storage-initializer
14:38:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj	Started	Started container storage-initializer
14:39:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb	Unhealthy	Readiness probe failed: Get "https://10.134.0.53:8643/healthz": dial tcp 10.134.0.53:8643: connect: connection refused
14:39:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1501" already present on machine
14:39:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj	Created	Created container: kube-rbac-proxy
14:39:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj	Created	Created container: kserve-container
14:39:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj	Started	Started container kserve-container
14:39:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:39:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj	Started	Started container kube-rbac-proxy
14:39:28 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:39:28 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:40:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	InferenceServiceReady	InferenceService [isvc-sklearn-v2] is Ready
14:40:11 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:40:13 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:40:13 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:40:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-v2-mixed": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-v2-mixed": the object has been modified; please apply your changes to the latest version and try again
14:40:18	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-mixed-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-mixed-predictor-7c7d4b6c98 from 0 to 1
14:40:19	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-mixed-predictor-7c7d4b6c98	SuccessfulCreate	Created pod: isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9
14:40:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj	Killing	Stopping container kube-rbac-proxy
14:40:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:40:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj	Killing	Stopping container kserve-container
14:40:19	kserve-ci-e2e-test	multus	isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9	AddedInterface	Add eth0 [10.134.0.55/23] from ovn-kubernetes
14:40:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9	Started	Started container storage-initializer
14:40:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9	Created	Created container: storage-initializer
14:40:21 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj	Unhealthy	Readiness probe failed: dial tcp 10.134.0.54:8080: connect: connection refused
14:40:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj	Unhealthy	Readiness probe failed: Get "https://10.134.0.54:8643/healthz": dial tcp 10.134.0.54:8643: connect: connection refused
14:40:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9	Created	Created container: kserve-container
14:40:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9	Started	Started container kube-rbac-proxy
14:40:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9	Created	Created container: kube-rbac-proxy
14:40:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:40:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9	Started	Started container kserve-container
14:40:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1501" already present on machine
14:40:33	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:40:34	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:41:31 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:41:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	InferenceServiceReady	InferenceService [isvc-sklearn-v2-mixed] is Ready
14:41:34 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:41:34 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:41:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9	Killing	Stopping container kube-rbac-proxy
14:41:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9	Killing	Stopping container kserve-container
14:41:39	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-x2hg5	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-tensorflow-predictor-serving-cert" not found
14:41:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow": the object has been modified; please apply your changes to the latest version and try again
14:41:39	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-predictor-6756f669d7 from 0 to 1
14:41:39	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-predictor-6756f669d7	SuccessfulCreate	Created pod: isvc-tensorflow-predictor-6756f669d7-x2hg5
14:41:39 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow": the object has been modified; please apply your changes to the latest version and try again
14:41:40	kserve-ci-e2e-test	multus	isvc-tensorflow-predictor-6756f669d7-x2hg5	AddedInterface	Add eth0 [10.134.0.56/23] from ovn-kubernetes
14:41:40	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-x2hg5	Started	Started container storage-initializer
14:41:40	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-x2hg5	Created	Created container: storage-initializer
14:41:40	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-x2hg5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:41:41 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9	Unhealthy	Readiness probe failed: dial tcp 10.134.0.55:8080: connect: connection refused
14:41:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9	Unhealthy	Readiness probe failed: Get "https://10.134.0.55:8643/healthz": dial tcp 10.134.0.55:8643: connect: connection refused
14:41:44	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-x2hg5	Pulling	Pulling image "tensorflow/serving:2.6.2"
14:41:48	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-x2hg5	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:41:48	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-x2hg5	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 3.949s (3.949s including waiting). Image size: 425873876 bytes.
14:41:48	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-x2hg5	Created	Created container: kserve-container
14:41:48	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-x2hg5	Started	Started container kserve-container
14:41:49	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-x2hg5	Created	Created container: kube-rbac-proxy
14:41:49	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-x2hg5	Started	Started container kube-rbac-proxy
14:41:56 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-x2hg5	Unhealthy	Readiness probe failed: dial tcp 10.134.0.56:8080: connect: connection refused
14:42:06	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	InferenceServiceReady	InferenceService [isvc-tensorflow] is Ready
14:42:06 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:42:09 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:42:09 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:42:19	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-vc58h	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f"
14:42:19	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-x2hg5	Killing	Stopping container kserve-container
14:42:19	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-runtime-predictor-8699d78cf	SuccessfulCreate	Created pod: isvc-tensorflow-runtime-predictor-8699d78cf-vc58h
14:42:19	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-runtime-predictor-8699d78cf from 0 to 1
14:42:19	kserve-ci-e2e-test	multus	isvc-tensorflow-runtime-predictor-8699d78cf-vc58h	AddedInterface	Add eth0 [10.133.0.37/23] from ovn-kubernetes
14:42:19	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-x2hg5	Killing	Stopping container kube-rbac-proxy
14:42:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-tensorflow-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-tensorflow-runtime": the object has been modified; please apply your changes to the latest version and try again
14:42:23	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-vc58h	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" in 4.079s (4.079s including waiting). Image size: 301469265 bytes.
14:42:23	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-vc58h	Created	Created container: storage-initializer
14:42:23	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-vc58h	Started	Started container storage-initializer
14:42:28	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-vc58h	Pulling	Pulling image "tensorflow/serving:2.6.2"
14:42:32	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-vc58h	Created	Created container: kserve-container
14:42:32	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-vc58h	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 3.555s (3.555s including waiting). Image size: 425873876 bytes.
14:42:32	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-vc58h	Started	Started container kserve-container
14:42:32	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-vc58h	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:42:32	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-vc58h	Created	Created container: kube-rbac-proxy
14:42:32	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-vc58h	Started	Started container kube-rbac-proxy
14:42:39 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-vc58h	Unhealthy	Readiness probe failed: dial tcp 10.133.0.37:8080: connect: connection refused
14:42:46 (x6)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-x2hg5	Unhealthy	Readiness probe failed: Get "https://10.134.0.56:8643/healthz": dial tcp 10.134.0.56:8643: connect: connection refused
14:42:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	InferenceServiceReady	InferenceService [isvc-tensorflow-runtime] is Ready
14:42:50 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:43:04 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:43:04 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:43:10	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-t6ffb	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:43:10	kserve-ci-e2e-test	multus	isvc-triton-predictor-84bb65d94b-t6ffb	AddedInterface	Add eth0 [10.133.0.38/23] from ovn-kubernetes
14:43:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	UpdateFailed	Failed to update status for InferenceService "isvc-triton": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-triton": the object has been modified; please apply your changes to the latest version and try again
14:43:10	kserve-ci-e2e-test	replicaset-controller	isvc-triton-predictor-84bb65d94b	SuccessfulCreate	Created pod: isvc-triton-predictor-84bb65d94b-t6ffb
14:43:10	kserve-ci-e2e-test	deployment-controller	isvc-triton-predictor	ScalingReplicaSet	Scaled up replica set isvc-triton-predictor-84bb65d94b from 0 to 1
14:43:10	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-vc58h	Killing	Stopping container kube-rbac-proxy
14:43:10	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-vc58h	Killing	Stopping container kserve-container
14:43:11	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-t6ffb	Started	Started container storage-initializer
14:43:11	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-t6ffb	Created	Created container: storage-initializer
14:43:14	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-t6ffb	Pulling	Pulling image "nvcr.io/nvidia/tritonserver:23.05-py3"
14:43:39 (x6)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-vc58h	Unhealthy	Readiness probe failed: Get "https://10.133.0.37:8643/healthz": dial tcp 10.133.0.37:8643: connect: connection refused
14:45:09	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-t6ffb	Started	Started container kserve-container
14:45:09	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-t6ffb	Pulled	Successfully pulled image "nvcr.io/nvidia/tritonserver:23.05-py3" in 1m54.168s (1m54.168s including waiting). Image size: 12907074623 bytes.
14:45:09	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-t6ffb	Started	Started container kube-rbac-proxy
14:45:09	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-t6ffb	Created	Created container: kube-rbac-proxy
14:45:09	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-t6ffb	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:45:09	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-t6ffb	Created	Created container: kserve-container
14:45:10 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:45:10 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:45:15 (x3)	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-t6ffb	Unhealthy	Readiness probe failed: dial tcp 10.133.0.38:8080: connect: connection refused
14:45:25 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:45:25	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	InferenceServiceReady	InferenceService [isvc-triton] is Ready
14:45:25	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:45:25	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:45:32 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost": the object has been modified; please apply your changes to the latest version and try again
14:45:32	kserve-ci-e2e-test	multus	isvc-xgboost-predictor-8689c4cfcc-8vsc7	AddedInterface	Add eth0 [10.133.0.39/23] from ovn-kubernetes
14:45:32	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-predictor-8689c4cfcc	SuccessfulCreate	Created pod: isvc-xgboost-predictor-8689c4cfcc-8vsc7
14:45:32	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-predictor-8689c4cfcc from 0 to 1
14:45:32	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-8vsc7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:45:32	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-t6ffb	Killing	Stopping container kube-rbac-proxy
14:45:32	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-t6ffb	Killing	Stopping container kserve-container
14:45:32	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost": the object has been modified; please apply your changes to the latest version and try again
14:45:33	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-8vsc7	Started	Started container storage-initializer
14:45:33	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-8vsc7	Created	Created container: storage-initializer
14:45:37	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-8vsc7	Pulling	Pulling image "kserve/xgbserver:latest"
14:45:57	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-8vsc7	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 20.419s (20.419s including waiting). Image size: 1306444539 bytes.
14:45:58	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-8vsc7	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:45:58	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-8vsc7	Started	Started container kserve-container
14:45:58	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-8vsc7	Created	Created container: kserve-container
14:45:58	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-8vsc7	Created	Created container: kube-rbac-proxy
14:45:58	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-8vsc7	Started	Started container kube-rbac-proxy
14:46:32 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:46:32 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:47:02 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:47:02 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:47:04 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:47:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	InferenceServiceReady	InferenceService [isvc-xgboost] is Ready
14:47:12	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-8vsc7	Killing	Stopping container kube-rbac-proxy
14:47:12 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
14:47:12	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
14:47:12	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-8vsc7	Killing	Stopping container kserve-container
14:47:12	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-mlserver-predictor-67d4bc6646 from 0 to 1
14:47:12	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-mlserver-predictor-67d4bc6646	SuccessfulCreate	Created pod: isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767
14:47:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:47:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767	Created	Created container: storage-initializer
14:47:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767	Started	Started container storage-initializer
14:47:13	kserve-ci-e2e-test	multus	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767	AddedInterface	Add eth0 [10.134.0.57/23] from ovn-kubernetes
14:47:14	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-8vsc7	Unhealthy	Readiness probe failed: Get "https://10.133.0.39:8643/healthz": dial tcp 10.133.0.39:8643: connect: connection refused
14:47:14 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-8vsc7	Unhealthy	Readiness probe failed: dial tcp 10.133.0.39:8080: connect: connection refused
14:47:17	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767	Started	Started container kserve-container
14:47:17	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:47:17	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767	Created	Created container: kube-rbac-proxy
14:47:17	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767	Created	Created container: kserve-container
14:47:17	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
14:47:17	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767	Started	Started container kube-rbac-proxy
14:47:27	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:47:27	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:47:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	InferenceServiceReady	InferenceService [isvc-xgboost-v2-mlserver] is Ready
14:47:55 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:47:57 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:47:57 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:48:02	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767	Killing	Stopping container kserve-container
14:48:02	kserve-ci-e2e-test	replicaset-controller	xgboost-v2-mlserver-predictor-7799869d6f	SuccessfulCreate	Created pod: xgboost-v2-mlserver-predictor-7799869d6f-cldbs
14:48:02	kserve-ci-e2e-test	deployment-controller	xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set xgboost-v2-mlserver-predictor-7799869d6f from 0 to 1
14:48:02	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767	Killing	Stopping container kube-rbac-proxy
14:48:02	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
14:48:03	kserve-ci-e2e-test	multus	xgboost-v2-mlserver-predictor-7799869d6f-cldbs	AddedInterface	Add eth0 [10.134.0.58/23] from ovn-kubernetes
14:48:03	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-cldbs	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "xgboost-v2-mlserver-predictor-serving-cert" not found
14:48:03	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-cldbs	Started	Started container storage-initializer
14:48:03	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-cldbs	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:48:03	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-cldbs	Created	Created container: storage-initializer
14:48:05	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767	Unhealthy	Readiness probe failed: Get "https://10.134.0.57:8643/healthz": dial tcp 10.134.0.57:8643: connect: connection refused
14:48:05	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767	Unhealthy	Readiness probe failed: Get "http://10.134.0.57:8080/v2/models/isvc-xgboost-v2-mlserver/ready": dial tcp 10.134.0.57:8080: connect: connection refused
14:48:08	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-cldbs	Started	Started container kube-rbac-proxy
14:48:08	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-cldbs	Created	Created container: kube-rbac-proxy
14:48:08	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-cldbs	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:48:08	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-cldbs	Started	Started container kserve-container
14:48:08	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-cldbs	Created	Created container: kserve-container
14:48:08	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-cldbs	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
14:48:46 (x13)	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:48:46	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	InferenceServiceReady	InferenceService [xgboost-v2-mlserver] is Ready
14:48:47 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:48:47 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:48:52	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-cldbs	Killing	Stopping container kube-rbac-proxy
14:48:52	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-cldbs	Killing	Stopping container kserve-container
14:48:53	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-m4wj7	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
14:48:53	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-runtime-predictor-779db84d9	SuccessfulCreate	Created pod: isvc-xgboost-runtime-predictor-779db84d9-m4wj7
14:48:53	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-runtime-predictor-779db84d9 from 0 to 1
14:48:53 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-runtime": the object has been modified; please apply your changes to the latest version and try again
14:48:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-runtime": the object has been modified; please apply your changes to the latest version and try again
14:48:54	kserve-ci-e2e-test	multus	isvc-xgboost-runtime-predictor-779db84d9-m4wj7	AddedInterface	Add eth0 [10.133.0.40/23] from ovn-kubernetes
14:48:54	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-m4wj7	Started	Started container storage-initializer
14:48:54	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-m4wj7	Created	Created container: storage-initializer
14:48:54	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-m4wj7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:48:56	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-cldbs	Unhealthy	Readiness probe failed: Get "http://10.134.0.58:8080/v2/models/xgboost-v2-mlserver/ready": dial tcp 10.134.0.58:8080: connect: connection refused
14:48:56	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-cldbs	Unhealthy	Readiness probe failed: Get "https://10.134.0.58:8643/healthz": dial tcp 10.134.0.58:8643: connect: connection refused
14:48:58	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-m4wj7	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:48:58	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-m4wj7	Started	Started container kserve-container
14:48:58	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-m4wj7	Created	Created container: kserve-container
14:48:58	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-m4wj7	Pulled	Container image "kserve/xgbserver:latest" already present on machine
14:48:58	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-m4wj7	Created	Created container: kube-rbac-proxy
14:48:58	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-m4wj7	Started	Started container kube-rbac-proxy
14:49:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:49:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:50:06 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:50:06	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-runtime] is Ready
14:50:08 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:50:08 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:50:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:50:13	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-runtime-predictor-6dc5954dc	SuccessfulCreate	Created pod: isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh
14:50:13	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-runtime-predictor-6dc5954dc from 0 to 1
14:50:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-m4wj7	Killing	Stopping container kserve-container
14:50:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh	Started	Started container storage-initializer
14:50:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh	Created	Created container: storage-initializer
14:50:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-m4wj7	Killing	Stopping container kube-rbac-proxy
14:50:13	kserve-ci-e2e-test	multus	isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh	AddedInterface	Add eth0 [10.134.0.59/23] from ovn-kubernetes
14:50:13 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
14:50:13	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
14:50:16 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-m4wj7	Unhealthy	Readiness probe failed: dial tcp 10.133.0.40:8080: connect: connection refused
14:50:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-m4wj7	Unhealthy	Readiness probe failed: Get "https://10.133.0.40:8643/healthz": dial tcp 10.133.0.40:8643: connect: connection refused
14:50:17	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:50:17	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh	Created	Created container: kserve-container
14:50:17	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
14:50:17	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh	Started	Started container kserve-container
14:50:17	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh	Created	Created container: kube-rbac-proxy
14:50:17	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh	Started	Started container kube-rbac-proxy
14:50:28	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:50:28	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:50:55	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
14:51:05	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-v2-runtime] is Ready
14:51:05 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:51:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh	Killing	Stopping container kserve-container
14:51:13	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
14:51:13 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:51:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh	Killing	Stopping container kube-rbac-proxy
14:51:13	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-predictor-6fcdd6977c from 0 to 1
14:51:13	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-predictor-6fcdd6977c	SuccessfulCreate	Created pod: isvc-xgboost-v2-predictor-6fcdd6977c-jkjm4
14:51:13 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:51:13	kserve-ci-e2e-test	multus	isvc-xgboost-v2-predictor-6fcdd6977c-jkjm4	AddedInterface	Add eth0 [10.133.0.41/23] from ovn-kubernetes
14:51:14	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-jkjm4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:51:14	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-jkjm4	Created	Created container: storage-initializer
14:51:14	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-jkjm4	Started	Started container storage-initializer
14:51:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh	Unhealthy	Readiness probe failed: Get "https://10.134.0.59:8643/healthz": dial tcp 10.134.0.59:8643: connect: connection refused
14:51:18	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-jkjm4	Pulled	Container image "kserve/xgbserver:latest" already present on machine
14:51:18	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-jkjm4	Created	Created container: kube-rbac-proxy
14:51:18	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-jkjm4	Created	Created container: kserve-container
14:51:18	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-jkjm4	Started	Started container kserve-container
14:51:18	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-jkjm4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:51:18	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-jkjm4	Started	Started container kube-rbac-proxy
14:51:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:51:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:52:25 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:52:25	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	InferenceServiceReady	InferenceService [isvc-xgboost-v2] is Ready
14:52:28 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:52:28 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:52:33	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-jkjm4	Killing	Stopping container kserve-container
14:52:33	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-jkjm4	Killing	Stopping container kube-rbac-proxy
14:52:33	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3": the object has been modified; please apply your changes to the latest version and try again
14:52:33	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-predictor-6c746f6458	SuccessfulCreate	Created pod: isvc-sklearn-s3-predictor-6c746f6458-zshfq
14:52:33	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-predictor-6c746f6458 from 0 to 1
14:52:34	kserve-ci-e2e-test	multus	isvc-sklearn-s3-predictor-6c746f6458-zshfq	AddedInterface	Add eth0 [10.134.0.60/23] from ovn-kubernetes
14:52:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c746f6458-zshfq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:52:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c746f6458-zshfq	Created	Created container: storage-initializer
14:52:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c746f6458-zshfq	Started	Started container storage-initializer
14:52:35 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-jkjm4	Unhealthy	Readiness probe failed: dial tcp 10.133.0.41:8080: connect: connection refused
14:52:35	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-jkjm4	Unhealthy	Readiness probe failed: Get "https://10.133.0.41:8643/healthz": dial tcp 10.133.0.41:8643: connect: connection refused
14:52:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c746f6458-zshfq	Created	Created container: kserve-container
14:52:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c746f6458-zshfq	Started	Started container kube-rbac-proxy
14:52:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c746f6458-zshfq	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1501" already present on machine
14:52:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c746f6458-zshfq	Created	Created container: kube-rbac-proxy
14:52:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c746f6458-zshfq	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:52:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c746f6458-zshfq	Started	Started container kserve-container
14:52:48	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:52:48	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:53:44 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:53:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	InferenceServiceReady	InferenceService [isvc-sklearn-s3] is Ready
14:53:48 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:53:48 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:53:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c746f6458-zshfq	Killing	Stopping container kube-rbac-proxy
14:53:53	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs
14:53:53	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd from 0 to 1
14:53:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-pass": the object has been modified; please apply your changes to the latest version and try again
14:53:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c746f6458-zshfq	Killing	Stopping container kserve-container
14:53:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c746f6458-zshfq	Unhealthy	Readiness probe failed: Get "https://10.134.0.60:8643/healthz": dial tcp 10.134.0.60:8643: connect: connection refused
14:53:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
14:53:54	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs	AddedInterface	Add eth0 [10.134.0.61/23] from ovn-kubernetes
14:53:54 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-6c746f6458-zshfq	Unhealthy	Readiness probe failed: dial tcp 10.134.0.60:8080: connect: connection refused
14:53:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:53:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs	Created	Created container: storage-initializer
14:53:55	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs	Started	Started container storage-initializer
14:53:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1501" already present on machine
14:53:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs	Started	Started container kube-rbac-proxy
14:53:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs	Created	Created container: kserve-container
14:53:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs	Started	Started container kserve-container
14:53:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:53:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs	Created	Created container: kube-rbac-proxy
14:54:08	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:54:08	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:55:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-global-pass] is Ready
14:55:04 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:55:08 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:55:08 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:55:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs	Killing	Stopping container kube-rbac-proxy
14:55:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs	Killing	Stopping container kserve-container
14:55:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs	Unhealthy	Readiness probe failed: Get "https://10.134.0.61:8643/healthz": dial tcp 10.134.0.61:8643: connect: connection refused
14:55:14 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs	Unhealthy	Readiness probe failed: dial tcp 10.134.0.61:8080: connect: connection refused
14:55:14	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b from 0 to 1
14:55:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-fail": the object has been modified; please apply your changes to the latest version and try again
14:55:15	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr
14:55:15 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-fail": the object has been modified; please apply your changes to the latest version and try again
14:55:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
14:55:16	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr	AddedInterface	Add eth0 [10.134.0.62/23] from ovn-kubernetes
14:55:21 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:55:21 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr	Started	Started container storage-initializer
14:55:21 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr	Created	Created container: storage-initializer
14:55:21 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:55:25	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr	Killing	Stopping container storage-initializer
14:55:26	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-pass": the object has been modified; please apply your changes to the latest version and try again
14:55:26	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b from 0 to 1
14:55:26	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v
14:55:26 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-pass": the object has been modified; please apply your changes to the latest version and try again
14:55:26	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v	AddedInterface	Add eth0 [10.134.0.63/23] from ovn-kubernetes
14:55:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:55:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v	Created	Created container: storage-initializer
14:55:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v	Started	Started container storage-initializer
14:55:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v	Created	Created container: kserve-container
14:55:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v	Started	Started container kserve-container
14:55:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1501" already present on machine
14:55:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:55:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v	Created	Created container: kube-rbac-proxy
14:55:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v	Started	Started container kube-rbac-proxy
14:55:56 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:55:56 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:56:25 (x8)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v	Unhealthy	Readiness probe failed: dial tcp 10.134.0.63:8080: connect: connection refused
14:56:26 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:56:26 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:56:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-custom-pass] is Ready
14:56:35 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:56:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v	Killing	Stopping container kserve-container
14:56:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v	Killing	Stopping container kube-rbac-proxy
14:56:37	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d from 0 to 1
14:56:37	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv
14:56:37	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-fail": the object has been modified; please apply your changes to the latest version and try again
14:56:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
14:56:38	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv	AddedInterface	Add eth0 [10.134.0.64/23] from ovn-kubernetes
14:56:44 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:56:45 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:56:45 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv	Created	Created container: storage-initializer
14:56:45 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv	Started	Started container storage-initializer
14:56:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv	Killing	Stopping container storage-initializer
14:56:48	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2
14:56:48 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-pass": the object has been modified; please apply your changes to the latest version and try again
14:56:48	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df from 0 to 1
14:56:48	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-pass": the object has been modified; please apply your changes to the latest version and try again
14:56:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
14:56:49	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2	AddedInterface	Add eth0 [10.134.0.65/23] from ovn-kubernetes
14:56:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2	Started	Started container storage-initializer
14:56:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2	Created	Created container: storage-initializer
14:56:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:56:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1501" already present on machine
14:56:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2	Created	Created container: kube-rbac-proxy
14:56:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2	Created	Created container: kserve-container
14:56:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2	Started	Started container kube-rbac-proxy
14:56:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
14:56:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2	Started	Started container kserve-container
14:57:33 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:57:33 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
14:57:59	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-serving-pass] is Ready
14:57:59 (x13)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:58:03 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:58:03 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
14:58:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2	Killing	Stopping container kserve-container
14:58:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2	Killing	Stopping container kube-rbac-proxy
14:58:09 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2	Unhealthy	Readiness probe failed: dial tcp 10.134.0.65:8080: connect: connection refused
14:58:09	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-fail": the object has been modified; please apply your changes to the latest version and try again
14:58:09	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5
14:58:09	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5	AddedInterface	Add eth0 [10.134.0.66/23] from ovn-kubernetes
14:58:09	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59 from 0 to 1
14:58:14 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2	Unhealthy	Readiness probe failed: Get "https://10.134.0.65:8643/healthz": dial tcp 10.134.0.65:8643: connect: connection refused
14:58:16 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5	Started	Started container storage-initializer
14:58:16 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5	Created	Created container: storage-initializer
14:58:16 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
14:58:16 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0459e31fff147b1f9487117ac00a86221572ce2251d3068d36cf76ed2126ea4f" already present on machine
14:58:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5	Killing	Stopping container storage-initializer