Events

Time	Namespace	Component	RelatedObject	Reason	Message
19:36:00	kserve-ci-e2e-test		isvc-sklearn-mcp-predictor-5f8b5bfcd6-nw5gq	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f8b5bfcd6-nw5gq to ip-10-0-139-128.ec2.internal
19:38:31	kserve-ci-e2e-test		isvc-paddle-runtime-predictor-7f4d4f9dc8-9thsb	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-9thsb to ip-10-0-139-128.ec2.internal
19:59:48	kserve-ci-e2e-test		isvc-sklearn-v2-runtime-predictor-6d84c876f4-8ctdr	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8ctdr to ip-10-0-139-128.ec2.internal
19:52:18	kserve-ci-e2e-test		isvc-predictive-sklearn-v2-predictor-b5d4f6b79-5l4f9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-5l4f9 to ip-10-0-143-206.ec2.internal
20:02:09	kserve-ci-e2e-test		isvc-sklearn-v2-mixed-predictor-86d7579fd6-87x9p	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-86d7579fd6-87x9p to ip-10-0-139-128.ec2.internal
19:42:16	kserve-ci-e2e-test		isvc-pmml-runtime-predictor-67bc544947-2zztw	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-2zztw to ip-10-0-143-206.ec2.internal
19:25:04	kserve-ci-e2e-test		isvc-sklearn-batcher-custom-predictor-7c59ff5d-hd5lz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7c59ff5d-hd5lz to ip-10-0-139-128.ec2.internal
19:46:52	kserve-ci-e2e-test		isvc-secondary-77d500-predictor-7cbd677c59-wpc97	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-secondary-77d500-predictor-7cbd677c59-wpc97 to ip-10-0-139-128.ec2.internal
19:45:33	kserve-ci-e2e-test		isvc-primary-77d500-predictor-8d9ffc784-l4s82	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-primary-77d500-predictor-8d9ffc784-l4s82 to ip-10-0-139-128.ec2.internal
20:00:49	kserve-ci-e2e-test		isvc-sklearn-v2-predictor-64fcb8589f-wgmk5	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64fcb8589f-wgmk5 to ip-10-0-139-128.ec2.internal
19:34:39	kserve-ci-e2e-test		isvc-lightgbm-v2-kserve-predictor-559bf6989-t4pcr	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-t4pcr to ip-10-0-139-128.ec2.internal
19:28:14	kserve-ci-e2e-test		isvc-lightgbm-predictor-bdf964bd-ltxg9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-ltxg9 to ip-10-0-139-128.ec2.internal
19:47:19	kserve-ci-e2e-test		isvc-predictive-sklearn-predictor-cd7c759c9-54b7w	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-54b7w to ip-10-0-143-206.ec2.internal
19:47:09	kserve-ci-e2e-test		isvc-init-fail-19fe47-predictor-c5688fb5c-bh8pm	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-init-fail-19fe47-predictor-c5688fb5c-bh8pm to ip-10-0-139-128.ec2.internal
19:35:09	kserve-ci-e2e-test		isvc-mlflow-v2-runtime-predictor-5fdb47d546-g52mx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-g52mx to ip-10-0-139-128.ec2.internal
19:37:20	kserve-ci-e2e-test		isvc-paddle-predictor-6b8b7cfb4b-ts72q	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-ts72q to ip-10-0-139-128.ec2.internal
19:49:14	kserve-ci-e2e-test		isvc-predictive-xgboost-predictor-7ff98fd74d-x6gsl	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-x6gsl to ip-10-0-143-206.ec2.internal
19:40:34	kserve-ci-e2e-test		isvc-pmml-predictor-8bb578669-zzb69	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-zzb69 to ip-10-0-139-128.ec2.internal
19:44:02	kserve-ci-e2e-test		isvc-pmml-v2-kserve-predictor-6578f8ffc7-cx7b9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-cx7b9 to ip-10-0-143-206.ec2.internal
19:26:49	kserve-ci-e2e-test		isvc-logger-predictor-7d4db54646-dlmkf	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-logger-predictor-7d4db54646-dlmkf to ip-10-0-139-128.ec2.internal
19:26:29	kserve-ci-e2e-test		message-dumper-predictor-c7d86bcbd-x8c6j	Scheduled	Successfully assigned kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-x8c6j to ip-10-0-139-128.ec2.internal
20:11:48	kserve-ci-e2e-test		isvc-xgboost-v2-runtime-predictor-6dc5954dc-sp5z2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-sp5z2 to ip-10-0-139-128.ec2.internal
20:12:49	kserve-ci-e2e-test		isvc-xgboost-v2-predictor-6fcdd6977c-k9kkk	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-k9kkk to ip-10-0-143-206.ec2.internal
19:39:33	kserve-ci-e2e-test		isvc-paddle-v2-kserve-predictor-7dbd59854-jphz7	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-jphz7 to ip-10-0-139-128.ec2.internal
20:08:48	kserve-ci-e2e-test		isvc-xgboost-v2-mlserver-predictor-67d4bc6646-cq8m2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-cq8m2 to ip-10-0-139-128.ec2.internal
20:10:28	kserve-ci-e2e-test		isvc-xgboost-runtime-predictor-779db84d9-w5ldr	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-w5ldr to ip-10-0-143-206.ec2.internal
20:07:08	kserve-ci-e2e-test		isvc-xgboost-predictor-8689c4cfcc-w8f78	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-w8f78 to ip-10-0-143-206.ec2.internal
20:04:56	kserve-ci-e2e-test		isvc-triton-predictor-84bb65d94b-jvtld	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-jvtld to ip-10-0-143-206.ec2.internal
20:04:14	kserve-ci-e2e-test		isvc-tensorflow-runtime-predictor-8699d78cf-fh7l5	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-fh7l5 to ip-10-0-143-206.ec2.internal
20:03:29	kserve-ci-e2e-test		isvc-tensorflow-predictor-6756f669d7-wpjsw	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-wpjsw to ip-10-0-143-206.ec2.internal
19:58:11	kserve-ci-e2e-test		sklearn-v2-mlserver-predictor-65d8664766-mw5mr	Scheduled	Successfully assigned kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-mw5mr to ip-10-0-139-128.ec2.internal
19:53:48	kserve-ci-e2e-test		isvc-predictive-xgboost-v2-predictor-6577c65fd8-svdg6	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-svdg6 to ip-10-0-143-206.ec2.internal
19:23:18	kserve-ci-e2e-test		isvc-sklearn-batcher-predictor-57fcff47c9-zrm52	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57fcff47c9-zrm52 to ip-10-0-139-128.ec2.internal
19:29:55	kserve-ci-e2e-test		isvc-lightgbm-runtime-predictor-749c4f6d58-67zf9	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-67zf9 to ip-10-0-139-128.ec2.internal
20:18:23	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-pass-predictor-545f7c7957-66n4x	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-545f7c7957-66n4x to ip-10-0-139-128.ec2.internal
20:19:34	kserve-ci-e2e-test		isvc-sklearn-s3-tls-serving-fail-predictor-7b6b7fcbd7-77h4d	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7b6b7fcbd7-77h4d to ip-10-0-139-128.ec2.internal
20:15:29	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-pass-predictor-764bc9bb86-5qmr4	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-764bc9bb86-5qmr4 to ip-10-0-139-128.ec2.internal
20:16:39	kserve-ci-e2e-test		isvc-sklearn-s3-tls-global-fail-predictor-8cfdd8b8d-nx7ll	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8cfdd8b8d-nx7ll to ip-10-0-139-128.ec2.internal
20:16:51	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-pass-predictor-848ff9798-tm765	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-848ff9798-tm765 to ip-10-0-139-128.ec2.internal
20:18:12	kserve-ci-e2e-test		isvc-sklearn-s3-tls-custom-fail-predictor-c7d959f56-zkxbc	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c7d959f56-zkxbc to ip-10-0-139-128.ec2.internal
20:14:09	kserve-ci-e2e-test		isvc-sklearn-s3-predictor-584b446894-dhtlj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-s3-predictor-584b446894-dhtlj to ip-10-0-139-128.ec2.internal
19:59:11	kserve-ci-e2e-test		isvc-sklearn-runtime-predictor-7c6499f57-hk2ls	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7c6499f57-hk2ls to ip-10-0-139-128.ec2.internal
19:56:50	kserve-ci-e2e-test		isvc-sklearn-predictor-6875c879b7-96mbk	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-predictor-6875c879b7-96mbk to ip-10-0-139-128.ec2.internal
19:50:48	kserve-ci-e2e-test		isvc-predictive-lightgbm-predictor-75cb94f9f-8whhc	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-8whhc to ip-10-0-143-206.ec2.internal
19:55:19	kserve-ci-e2e-test		isvc-predictive-lightgbm-v2-predictor-865b4598f7-ftgmz	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-ftgmz to ip-10-0-143-206.ec2.internal
19:31:25	kserve-ci-e2e-test		isvc-lightgbm-v2-runtime-predictor-8765c9667-z2x9p	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-z2x9p to ip-10-0-139-128.ec2.internal
20:09:38	kserve-ci-e2e-test		xgboost-v2-mlserver-predictor-7799869d6f-v4hvs	Scheduled	Successfully assigned kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-v4hvs to ip-10-0-139-128.ec2.internal
19:23:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
19:23:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
19:23:18	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-predictor-57fcff47c9	SuccessfulCreate	Created pod: isvc-sklearn-batcher-predictor-57fcff47c9-zrm52
19:23:18	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-predictor-57fcff47c9-zrm52	AddedInterface	Add eth0 [10.134.0.25/23] from ovn-kubernetes
19:23:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-57fcff47c9-zrm52	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f"
19:23:18	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-predictor-57fcff47c9 from 0 to 1
19:23:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-57fcff47c9-zrm52	Created	Created container: storage-initializer
19:23:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-57fcff47c9-zrm52	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" in 4.542s (4.542s including waiting). Image size: 301485528 bytes.
19:23:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-57fcff47c9-zrm52	Started	Started container storage-initializer
19:23:25 (x23)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:23:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-57fcff47c9-zrm52	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1404"
19:23:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-57fcff47c9-zrm52	Created	Created container: kserve-container
19:23:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-57fcff47c9-zrm52	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
19:23:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-57fcff47c9-zrm52	Started	Started container kserve-container
19:23:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-57fcff47c9-zrm52	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1404" in 12.738s (12.738s including waiting). Image size: 1560922562 bytes.
19:23:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-57fcff47c9-zrm52	Started	Started container kube-rbac-proxy
19:23:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-57fcff47c9-zrm52	Pulling	Pulling image "quay.io/opendatahub/kserve-agent@sha256:de59d4f440abaeb1e71b5977a2145cdbe8db88ded8ac16ca09f179d82ba41738"
19:23:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-57fcff47c9-zrm52	Created	Created container: kube-rbac-proxy
19:23:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-57fcff47c9-zrm52	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.506s (2.506s including waiting). Image size: 211946088 bytes.
19:23:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-57fcff47c9-zrm52	Started	Started container agent
19:23:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-57fcff47c9-zrm52	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:de59d4f440abaeb1e71b5977a2145cdbe8db88ded8ac16ca09f179d82ba41738" in 2.912s (2.912s including waiting). Image size: 238051450 bytes.
19:23:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-57fcff47c9-zrm52	Created	Created container: agent
19:24:18 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:24:18 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:24:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InferenceServiceReady	InferenceService [isvc-sklearn-batcher] is Ready
19:25:03	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-custom-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-custom-predictor-7c59ff5d from 0 to 1
19:25:03	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher-custom": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher-custom": the object has been modified; please apply your changes to the latest version and try again
19:25:03 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:25:03 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:25:04	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-custom-predictor-7c59ff5d-hd5lz	AddedInterface	Add eth0 [10.134.0.26/23] from ovn-kubernetes
19:25:04	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-custom-predictor-7c59ff5d	SuccessfulCreate	Created pod: isvc-sklearn-batcher-custom-predictor-7c59ff5d-hd5lz
19:25:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-7c59ff5d-hd5lz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:25:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-7c59ff5d-hd5lz	Created	Created container: storage-initializer
19:25:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-7c59ff5d-hd5lz	Started	Started container storage-initializer
19:25:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-57fcff47c9-zrm52	Killing	Stopping container kube-rbac-proxy
19:25:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-57fcff47c9-zrm52	Killing	Stopping container agent
19:25:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-57fcff47c9-zrm52	Killing	Stopping container kserve-container
19:25:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-7c59ff5d-hd5lz	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:de59d4f440abaeb1e71b5977a2145cdbe8db88ded8ac16ca09f179d82ba41738" already present on machine
19:25:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-7c59ff5d-hd5lz	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1404" already present on machine
19:25:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-7c59ff5d-hd5lz	Created	Created container: kserve-container
19:25:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-7c59ff5d-hd5lz	Started	Started container kserve-container
19:25:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-7c59ff5d-hd5lz	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:25:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-7c59ff5d-hd5lz	Started	Started container agent
19:25:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-7c59ff5d-hd5lz	Created	Created container: agent
19:25:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-7c59ff5d-hd5lz	Created	Created container: kube-rbac-proxy
19:25:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-7c59ff5d-hd5lz	Started	Started container kube-rbac-proxy
19:25:12 (x24)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:25:19 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-57fcff47c9-zrm52	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
19:25:19 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-57fcff47c9-zrm52	Unhealthy	Readiness probe failed: dial tcp 10.134.0.25:8080: connect: connection refused
19:25:27 (x5)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-57fcff47c9-zrm52	Unhealthy	Readiness probe failed: Get "https://10.134.0.25:8643/healthz": dial tcp 10.134.0.25:8643: connect: connection refused
19:25:48 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:25:48 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:26:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	InferenceServiceReady	InferenceService [isvc-sklearn-batcher-custom] is Ready
19:26:19 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:26:19 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:26:29	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-x8c6j	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "message-dumper-predictor-serving-cert" not found
19:26:29	kserve-ci-e2e-test	replicaset-controller	message-dumper-predictor-c7d86bcbd	SuccessfulCreate	Created pod: message-dumper-predictor-c7d86bcbd-x8c6j
19:26:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-7c59ff5d-hd5lz	Killing	Stopping container kserve-container
19:26:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-7c59ff5d-hd5lz	Killing	Stopping container kube-rbac-proxy
19:26:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-7c59ff5d-hd5lz	Killing	Stopping container agent
19:26:29	kserve-ci-e2e-test	deployment-controller	message-dumper-predictor	ScalingReplicaSet	Scaled up replica set message-dumper-predictor-c7d86bcbd from 0 to 1
19:26:30	kserve-ci-e2e-test	multus	message-dumper-predictor-c7d86bcbd-x8c6j	AddedInterface	Add eth0 [10.134.0.27/23] from ovn-kubernetes
19:26:30	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-x8c6j	Pulling	Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display"
19:26:31	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-x8c6j	Started	Started container kube-rbac-proxy
19:26:31	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-x8c6j	Created	Created container: kube-rbac-proxy
19:26:31	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-x8c6j	Started	Started container kserve-container
19:26:31	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-x8c6j	Created	Created container: kserve-container
19:26:31	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-x8c6j	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:26:31	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-x8c6j	Pulled	Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.225s (1.225s including waiting). Image size: 14813193 bytes.
19:26:37 (x25)	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:26:40	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	InferenceServiceReady	InferenceService [message-dumper] is Ready
19:26:45 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-7c59ff5d-hd5lz	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
19:26:45 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-7c59ff5d-hd5lz	Unhealthy	Readiness probe failed: dial tcp 10.134.0.26:5000: connect: connection refused
19:26:49	kserve-ci-e2e-test	deployment-controller	isvc-logger-predictor	ScalingReplicaSet	Scaled up replica set isvc-logger-predictor-7d4db54646 from 0 to 1
19:26:49	kserve-ci-e2e-test	replicaset-controller	isvc-logger-predictor-7d4db54646	SuccessfulCreate	Created pod: isvc-logger-predictor-7d4db54646-dlmkf
19:26:50	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7d4db54646-dlmkf	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:26:50	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7d4db54646-dlmkf	Started	Started container storage-initializer
19:26:50 (x5)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-7c59ff5d-hd5lz	Unhealthy	Readiness probe failed: Get "https://10.134.0.26:8643/healthz": dial tcp 10.134.0.26:8643: connect: connection refused
19:26:50	kserve-ci-e2e-test	multus	isvc-logger-predictor-7d4db54646-dlmkf	AddedInterface	Add eth0 [10.134.0.28/23] from ovn-kubernetes
19:26:50	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7d4db54646-dlmkf	Created	Created container: storage-initializer
19:26:54	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7d4db54646-dlmkf	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1404" already present on machine
19:26:54	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7d4db54646-dlmkf	Started	Started container kserve-container
19:26:54	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7d4db54646-dlmkf	Created	Created container: kserve-container
19:26:54	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7d4db54646-dlmkf	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:26:54	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7d4db54646-dlmkf	Created	Created container: kube-rbac-proxy
19:26:54	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7d4db54646-dlmkf	Started	Started container kube-rbac-proxy
19:26:54	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7d4db54646-dlmkf	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:de59d4f440abaeb1e71b5977a2145cdbe8db88ded8ac16ca09f179d82ba41738" already present on machine
19:26:54	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7d4db54646-dlmkf	Created	Created container: agent
19:26:54	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7d4db54646-dlmkf	Started	Started container agent
19:26:59 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:26:59 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:27:05 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:27:34 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:27:34 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:28:02	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	InferenceServiceReady	InferenceService [isvc-logger] is Ready
19:28:04 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:28:04 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:28:14	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-predictor-bdf964bd from 0 to 1
19:28:14	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7d4db54646-dlmkf	Killing	Stopping container agent
19:28:14	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7d4db54646-dlmkf	Killing	Stopping container kube-rbac-proxy
19:28:14	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-predictor-bdf964bd	SuccessfulCreate	Created pod: isvc-lightgbm-predictor-bdf964bd-ltxg9
19:28:14	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7d4db54646-dlmkf	Killing	Stopping container kserve-container
19:28:14	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-x8c6j	Killing	Stopping container kube-rbac-proxy
19:28:14	kserve-ci-e2e-test	kubelet	message-dumper-predictor-c7d86bcbd-x8c6j	Killing	Stopping container kserve-container
19:28:15	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-ltxg9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:28:15	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-ltxg9	Started	Started container storage-initializer
19:28:15	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-ltxg9	Created	Created container: storage-initializer
19:28:15	kserve-ci-e2e-test	multus	isvc-lightgbm-predictor-bdf964bd-ltxg9	AddedInterface	Add eth0 [10.134.0.29/23] from ovn-kubernetes
19:28:19	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-ltxg9	Pulling	Pulling image "kserve/lgbserver:latest"
19:28:23 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:28:26	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-ltxg9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:28:26	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-ltxg9	Started	Started container kube-rbac-proxy
19:28:26	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-ltxg9	Pulled	Successfully pulled image "kserve/lgbserver:latest" in 7.033s (7.033s including waiting). Image size: 606096651 bytes.
19:28:26	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-ltxg9	Created	Created container: kserve-container
19:28:26	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-ltxg9	Started	Started container kserve-container
19:28:26	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-ltxg9	Created	Created container: kube-rbac-proxy
19:28:32 (x11)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7d4db54646-dlmkf	Unhealthy	Readiness probe failed: dial tcp 10.134.0.28:8080: connect: connection refused
19:28:32 (x10)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7d4db54646-dlmkf	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
19:28:32 (x4)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-7d4db54646-dlmkf	Unhealthy	Readiness probe failed: Get "https://10.134.0.28:8643/healthz": dial tcp 10.134.0.28:8643: connect: connection refused
19:28:59 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:28:59 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:29:40 (x10)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-ltxg9	Unhealthy	Readiness probe failed: dial tcp 10.134.0.29:8080: connect: connection refused
19:29:44 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:29:44 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:29:51	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm	InferenceServiceReady	InferenceService [isvc-lightgbm] is Ready
19:29:54	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-ltxg9	Killing	Stopping container kserve-container
19:29:54	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-ltxg9	Killing	Stopping container kube-rbac-proxy
19:29:54	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-runtime-predictor-749c4f6d58 from 0 to 1
19:29:55	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-67zf9	Started	Started container storage-initializer
19:29:55	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-runtime-predictor-749c4f6d58	SuccessfulCreate	Created pod: isvc-lightgbm-runtime-predictor-749c4f6d58-67zf9
19:29:55	kserve-ci-e2e-test	multus	isvc-lightgbm-runtime-predictor-749c4f6d58-67zf9	AddedInterface	Add eth0 [10.134.0.30/23] from ovn-kubernetes
19:29:55	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-67zf9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:29:55	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-67zf9	Created	Created container: storage-initializer
19:29:58	kserve-ci-e2e-test	kubelet	isvc-lightgbm-predictor-bdf964bd-ltxg9	Unhealthy	Readiness probe failed: Get "https://10.134.0.29:8643/healthz": dial tcp 10.134.0.29:8643: connect: connection refused
19:29:59	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-67zf9	Created	Created container: kube-rbac-proxy
19:29:59	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-67zf9	Pulled	Container image "kserve/lgbserver:latest" already present on machine
19:29:59	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-67zf9	Created	Created container: kserve-container
19:29:59	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-67zf9	Started	Started container kserve-container
19:29:59	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-67zf9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:29:59	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-67zf9	Started	Started container kube-rbac-proxy
19:30:03 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:30:24 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:30:24 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:31:17	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-runtime] is Ready
19:31:24 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:31:24 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:31:25	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-runtime-predictor-8765c9667 from 0 to 1
19:31:25	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-67zf9	Killing	Stopping container kserve-container
19:31:25	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-67zf9	Killing	Stopping container kube-rbac-proxy
19:31:25	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-runtime-predictor-8765c9667	SuccessfulCreate	Created pod: isvc-lightgbm-v2-runtime-predictor-8765c9667-z2x9p
19:31:25	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-z2x9p	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
19:31:26	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-z2x9p	Created	Created container: storage-initializer
19:31:26	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-runtime-predictor-8765c9667-z2x9p	AddedInterface	Add eth0 [10.134.0.31/23] from ovn-kubernetes
19:31:26	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-z2x9p	Started	Started container storage-initializer
19:31:26	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-z2x9p	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:31:27	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-67zf9	Unhealthy	Readiness probe failed: Get "https://10.134.0.30:8643/healthz": dial tcp 10.134.0.30:8643: connect: connection refused
19:31:27 (x10)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-runtime-predictor-749c4f6d58-67zf9	Unhealthy	Readiness probe failed: dial tcp 10.134.0.30:8080: connect: connection refused
19:31:31	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-z2x9p	Pulling	Pulling image "docker.io/seldonio/mlserver:1.7.1"
19:31:33 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:33:55	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-z2x9p	Started	Started container kserve-container
19:33:55	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-z2x9p	Created	Created container: kube-rbac-proxy
19:33:55	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-z2x9p	Started	Started container kube-rbac-proxy
19:33:55	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-z2x9p	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:33:55	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-z2x9p	Created	Created container: kserve-container
19:33:55	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-z2x9p	Pulled	Successfully pulled image "docker.io/seldonio/mlserver:1.7.1" in 2m24.657s (2m24.657s including waiting). Image size: 10890461297 bytes.
19:34:25 (x12)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:34:25 (x12)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:34:33	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-runtime	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-runtime] is Ready
19:34:39	kserve-ci-e2e-test	replicaset-controller	isvc-lightgbm-v2-kserve-predictor-559bf6989	SuccessfulCreate	Created pod: isvc-lightgbm-v2-kserve-predictor-559bf6989-t4pcr
19:34:39	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-z2x9p	Killing	Stopping container kube-rbac-proxy
19:34:39	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-runtime-predictor-8765c9667-z2x9p	Killing	Stopping container kserve-container
19:34:39	kserve-ci-e2e-test	deployment-controller	isvc-lightgbm-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-lightgbm-v2-kserve-predictor-559bf6989 from 0 to 1
19:34:40	kserve-ci-e2e-test	multus	isvc-lightgbm-v2-kserve-predictor-559bf6989-t4pcr	AddedInterface	Add eth0 [10.134.0.32/23] from ovn-kubernetes
19:34:40	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-t4pcr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:34:40	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-t4pcr	Created	Created container: storage-initializer
19:34:40	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-t4pcr	Started	Started container storage-initializer
19:34:44	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-t4pcr	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:34:44	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-t4pcr	Pulled	Container image "kserve/lgbserver:latest" already present on machine
19:34:44	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-t4pcr	Created	Created container: kserve-container
19:34:44	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-t4pcr	Started	Started container kube-rbac-proxy
19:34:44	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-t4pcr	Created	Created container: kube-rbac-proxy
19:34:44	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-t4pcr	Started	Started container kserve-container
19:34:47 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:34:51 (x3)	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-t4pcr	Unhealthy	Readiness probe failed: dial tcp 10.134.0.32:8080: connect: connection refused
19:34:54	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:34:54	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:35:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-lightgbm-v2-kserve	InferenceServiceReady	InferenceService [isvc-lightgbm-v2-kserve] is Ready
19:35:09	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:35:09	kserve-ci-e2e-test	replicaset-controller	isvc-mlflow-v2-runtime-predictor-5fdb47d546	SuccessfulCreate	Created pod: isvc-mlflow-v2-runtime-predictor-5fdb47d546-g52mx
19:35:09	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-lightgbm-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:35:09	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-t4pcr	Killing	Stopping container kube-rbac-proxy
19:35:09	kserve-ci-e2e-test	kubelet	isvc-lightgbm-v2-kserve-predictor-559bf6989-t4pcr	Killing	Stopping container kserve-container
19:35:09	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-mlflow-v2-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-mlflow-v2-runtime": the object has been modified; please apply your changes to the latest version and try again
19:35:09	kserve-ci-e2e-test	deployment-controller	isvc-mlflow-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-mlflow-v2-runtime-predictor-5fdb47d546 from 0 to 1
19:35:10	kserve-ci-e2e-test	multus	isvc-mlflow-v2-runtime-predictor-5fdb47d546-g52mx	AddedInterface	Add eth0 [10.134.0.33/23] from ovn-kubernetes
19:35:10	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-g52mx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:35:10	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-g52mx	Created	Created container: storage-initializer
19:35:10	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-g52mx	Started	Started container storage-initializer
19:35:14	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-g52mx	Started	Started container kserve-container
19:35:14	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-g52mx	Created	Created container: kserve-container
19:35:14	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-g52mx	Created	Created container: kube-rbac-proxy
19:35:14	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-g52mx	Started	Started container kube-rbac-proxy
19:35:14	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-g52mx	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:35:14	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-g52mx	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:35:17 (x24)	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:35:39 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:35:39 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:35:52	kserve-ci-e2e-test	v1beta1Controllers	isvc-mlflow-v2-runtime	InferenceServiceReady	InferenceService [isvc-mlflow-v2-runtime] is Ready
19:35:54	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:35:54	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-mlflow-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:35:59	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-mcp-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-mcp-predictor-5f8b5bfcd6 from 0 to 1
19:35:59	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-g52mx	Killing	Stopping container kube-rbac-proxy
19:35:59	kserve-ci-e2e-test	kubelet	isvc-mlflow-v2-runtime-predictor-5fdb47d546-g52mx	Killing	Stopping container kserve-container
19:36:00	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-mcp-predictor-5f8b5bfcd6	SuccessfulCreate	Created pod: isvc-sklearn-mcp-predictor-5f8b5bfcd6-nw5gq
19:36:00	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5f8b5bfcd6-nw5gq	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
19:36:01	kserve-ci-e2e-test	multus	isvc-sklearn-mcp-predictor-5f8b5bfcd6-nw5gq	AddedInterface	Add eth0 [10.134.0.34/23] from ovn-kubernetes
19:36:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5f8b5bfcd6-nw5gq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:36:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5f8b5bfcd6-nw5gq	Created	Created container: storage-initializer
19:36:01	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5f8b5bfcd6-nw5gq	Started	Started container storage-initializer
19:36:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5f8b5bfcd6-nw5gq	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1404" already present on machine
19:36:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5f8b5bfcd6-nw5gq	Created	Created container: kserve-container
19:36:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5f8b5bfcd6-nw5gq	Started	Started container kserve-container
19:36:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5f8b5bfcd6-nw5gq	Pulling	Pulling image "quay.io/opendatahub/kserve-agent:latest"
19:36:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5f8b5bfcd6-nw5gq	Created	Created container: kserve-agent
19:36:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5f8b5bfcd6-nw5gq	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent:latest" in 2.893s (2.893s including waiting). Image size: 237817895 bytes.
19:36:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5f8b5bfcd6-nw5gq	Started	Started container kserve-agent
19:36:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5f8b5bfcd6-nw5gq	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:36:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5f8b5bfcd6-nw5gq	Created	Created container: kube-rbac-proxy
19:36:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5f8b5bfcd6-nw5gq	Started	Started container kube-rbac-proxy
19:36:08 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:37:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-mcp	InferenceServiceReady	InferenceService [isvc-sklearn-mcp] is Ready
19:37:19	kserve-ci-e2e-test	deployment-controller	isvc-paddle-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-predictor-6b8b7cfb4b from 0 to 1
19:37:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5f8b5bfcd6-nw5gq	Killing	Stopping container kube-rbac-proxy
19:37:20	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-ts72q	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:37:20	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-predictor-6b8b7cfb4b	SuccessfulCreate	Created pod: isvc-paddle-predictor-6b8b7cfb4b-ts72q
19:37:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5f8b5bfcd6-nw5gq	Killing	Stopping container kserve-container
19:37:20	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-ts72q	Started	Started container storage-initializer
19:37:20	kserve-ci-e2e-test	multus	isvc-paddle-predictor-6b8b7cfb4b-ts72q	AddedInterface	Add eth0 [10.134.0.35/23] from ovn-kubernetes
19:37:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5f8b5bfcd6-nw5gq	Killing	Stopping container kserve-agent
19:37:20	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-ts72q	Created	Created container: storage-initializer
19:37:25	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-ts72q	Pulling	Pulling image "kserve/paddleserver:latest"
19:37:28 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:37:36	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-ts72q	Started	Started container kube-rbac-proxy
19:37:36	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-ts72q	Started	Started container kserve-container
19:37:36	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-ts72q	Created	Created container: kserve-container
19:37:36	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-ts72q	Pulled	Successfully pulled image "kserve/paddleserver:latest" in 10.842s (10.842s including waiting). Image size: 1162587384 bytes.
19:37:36	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-ts72q	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:37:36	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-ts72q	Created	Created container: kube-rbac-proxy
19:37:45 (x6)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5f8b5bfcd6-nw5gq	Unhealthy	Readiness probe failed: Get "https://10.134.0.34:8643/healthz": dial tcp 10.134.0.34:8643: connect: connection refused
19:37:45 (x4)	kserve-ci-e2e-test	kubelet	isvc-sklearn-mcp-predictor-5f8b5bfcd6-nw5gq	Unhealthy	Readiness probe failed: Get "http://10.134.0.34:8080/v1/models/isvc-sklearn-mcp": dial tcp 10.134.0.34:8080: connect: connection refused
19:38:04 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:38:04 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:38:19	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:38:19	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:38:23	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle	InferenceServiceReady	InferenceService [isvc-paddle] is Ready
19:38:31	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-runtime-predictor-7f4d4f9dc8	SuccessfulCreate	Created pod: isvc-paddle-runtime-predictor-7f4d4f9dc8-9thsb
19:38:31	kserve-ci-e2e-test	deployment-controller	isvc-paddle-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-runtime-predictor-7f4d4f9dc8 from 0 to 1
19:38:31	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-ts72q	Killing	Stopping container kube-rbac-proxy
19:38:31	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-ts72q	Killing	Stopping container kserve-container
19:38:32	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-9thsb	Created	Created container: storage-initializer
19:38:32	kserve-ci-e2e-test	multus	isvc-paddle-runtime-predictor-7f4d4f9dc8-9thsb	AddedInterface	Add eth0 [10.134.0.36/23] from ovn-kubernetes
19:38:32	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-9thsb	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:38:32	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-9thsb	Started	Started container storage-initializer
19:38:33 (x7)	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-ts72q	Unhealthy	Readiness probe failed: dial tcp 10.134.0.35:8080: connect: connection refused
19:38:33	kserve-ci-e2e-test	kubelet	isvc-paddle-predictor-6b8b7cfb4b-ts72q	Unhealthy	Readiness probe failed: Get "https://10.134.0.35:8643/healthz": dial tcp 10.134.0.35:8643: connect: connection refused
19:38:37	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-9thsb	Created	Created container: kube-rbac-proxy
19:38:37	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-9thsb	Created	Created container: kserve-container
19:38:37	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-9thsb	Pulled	Container image "kserve/paddleserver:latest" already present on machine
19:38:37	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-9thsb	Started	Started container kserve-container
19:38:37	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-9thsb	Started	Started container kube-rbac-proxy
19:38:37	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-9thsb	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:38:40 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:39:16 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:39:16 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:39:25	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-runtime	InferenceServiceReady	InferenceService [isvc-paddle-runtime] is Ready
19:39:31	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:39:31	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:39:32	kserve-ci-e2e-test	deployment-controller	isvc-paddle-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-paddle-v2-kserve-predictor-7dbd59854 from 0 to 1
19:39:33	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-jphz7	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:39:33	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-jphz7	Created	Created container: storage-initializer
19:39:33	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-9thsb	Killing	Stopping container kserve-container
19:39:33	kserve-ci-e2e-test	multus	isvc-paddle-v2-kserve-predictor-7dbd59854-jphz7	AddedInterface	Add eth0 [10.134.0.37/23] from ovn-kubernetes
19:39:33	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-jphz7	Started	Started container storage-initializer
19:39:33	kserve-ci-e2e-test	replicaset-controller	isvc-paddle-v2-kserve-predictor-7dbd59854	SuccessfulCreate	Created pod: isvc-paddle-v2-kserve-predictor-7dbd59854-jphz7
19:39:33	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-9thsb	Killing	Stopping container kube-rbac-proxy
19:39:35	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-9thsb	Unhealthy	Readiness probe failed: Get "https://10.134.0.36:8643/healthz": dial tcp 10.134.0.36:8643: connect: connection refused
19:39:35 (x7)	kserve-ci-e2e-test	kubelet	isvc-paddle-runtime-predictor-7f4d4f9dc8-9thsb	Unhealthy	Readiness probe failed: dial tcp 10.134.0.36:8080: connect: connection refused
19:39:38	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-jphz7	Started	Started container kserve-container
19:39:38	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-jphz7	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:39:38	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-jphz7	Created	Created container: kube-rbac-proxy
19:39:38	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-jphz7	Created	Created container: kserve-container
19:39:38	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-jphz7	Pulled	Container image "kserve/paddleserver:latest" already present on machine
19:39:38	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-jphz7	Started	Started container kube-rbac-proxy
19:39:41 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:39:48	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:39:48	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:40:26	kserve-ci-e2e-test	v1beta1Controllers	isvc-paddle-v2-kserve	InferenceServiceReady	InferenceService [isvc-paddle-v2-kserve] is Ready
19:40:33 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:40:33 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-paddle-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:40:34	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-jphz7	Killing	Stopping container kube-rbac-proxy
19:40:34	kserve-ci-e2e-test	deployment-controller	isvc-pmml-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-predictor-8bb578669 from 0 to 1
19:40:34	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-jphz7	Killing	Stopping container kserve-container
19:40:34	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-predictor-8bb578669	SuccessfulCreate	Created pod: isvc-pmml-predictor-8bb578669-zzb69
19:40:35	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-zzb69	Started	Started container storage-initializer
19:40:35	kserve-ci-e2e-test	multus	isvc-pmml-predictor-8bb578669-zzb69	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
19:40:35	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-zzb69	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:40:35	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-zzb69	Created	Created container: storage-initializer
19:40:36 (x7)	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-jphz7	Unhealthy	Readiness probe failed: dial tcp 10.134.0.37:8080: connect: connection refused
19:40:36	kserve-ci-e2e-test	kubelet	isvc-paddle-v2-kserve-predictor-7dbd59854-jphz7	Unhealthy	Readiness probe failed: Get "https://10.134.0.37:8643/healthz": dial tcp 10.134.0.37:8643: connect: connection refused
19:40:39	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-zzb69	Pulling	Pulling image "kserve/pmmlserver:latest"
19:40:43 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:40:46	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-zzb69	Started	Started container kserve-container
19:40:46	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-zzb69	Started	Started container kube-rbac-proxy
19:40:46	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-zzb69	Pulled	Successfully pulled image "kserve/pmmlserver:latest" in 6.88s (6.88s including waiting). Image size: 801068920 bytes.
19:40:46	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-zzb69	Created	Created container: kserve-container
19:40:46	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-zzb69	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:40:46	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-zzb69	Created	Created container: kube-rbac-proxy
19:41:19 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:41:19 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:42:02 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-zzb69	Unhealthy	Readiness probe failed: dial tcp 10.134.0.38:8080: connect: connection refused
19:42:04 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:42:04 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:42:13	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml	InferenceServiceReady	InferenceService [isvc-pmml] is Ready
19:42:16	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-zzb69	Killing	Stopping container kube-rbac-proxy
19:42:16	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-zzb69	Killing	Stopping container kserve-container
19:42:16	kserve-ci-e2e-test	deployment-controller	isvc-pmml-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-runtime-predictor-67bc544947 from 0 to 1
19:42:16	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-runtime-predictor-67bc544947	SuccessfulCreate	Created pod: isvc-pmml-runtime-predictor-67bc544947-2zztw
19:42:16	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	UpdateFailed	Failed to update status for InferenceService "isvc-pmml-runtime": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-pmml-runtime": the object has been modified; please apply your changes to the latest version and try again
19:42:17	kserve-ci-e2e-test	kubelet	isvc-pmml-predictor-8bb578669-zzb69	Unhealthy	Readiness probe failed: Get "https://10.134.0.38:8643/healthz": dial tcp 10.134.0.38:8643: connect: connection refused
19:42:17	kserve-ci-e2e-test	multus	isvc-pmml-runtime-predictor-67bc544947-2zztw	AddedInterface	Add eth0 [10.133.0.23/23] from ovn-kubernetes
19:42:17	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-2zztw	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f"
19:42:22	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-2zztw	Created	Created container: storage-initializer
19:42:22	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-2zztw	Started	Started container storage-initializer
19:42:22	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-2zztw	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" in 5.108s (5.108s including waiting). Image size: 301485528 bytes.
19:42:24 (x24)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:42:26	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-2zztw	Pulling	Pulling image "kserve/pmmlserver:latest"
19:42:33	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-2zztw	Pulled	Successfully pulled image "kserve/pmmlserver:latest" in 7.11s (7.11s including waiting). Image size: 801068920 bytes.
19:42:33	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-2zztw	Created	Created container: kserve-container
19:42:33	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-2zztw	Started	Started container kserve-container
19:42:33	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-2zztw	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
19:42:36	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-2zztw	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.408s (2.408s including waiting). Image size: 211946088 bytes.
19:42:36	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-2zztw	Created	Created container: kube-rbac-proxy
19:42:36	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-2zztw	Started	Started container kube-rbac-proxy
19:42:46 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:42:46 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:43:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-runtime	InferenceServiceReady	InferenceService [isvc-pmml-runtime] is Ready
19:44:01 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:44:01	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-2zztw	Killing	Stopping container kserve-container
19:44:01	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-2zztw	Killing	Stopping container kube-rbac-proxy
19:44:01 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:44:01	kserve-ci-e2e-test	deployment-controller	isvc-pmml-v2-kserve-predictor	ScalingReplicaSet	Scaled up replica set isvc-pmml-v2-kserve-predictor-6578f8ffc7 from 0 to 1
19:44:02	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-cx7b9	Started	Started container storage-initializer
19:44:02	kserve-ci-e2e-test	replicaset-controller	isvc-pmml-v2-kserve-predictor-6578f8ffc7	SuccessfulCreate	Created pod: isvc-pmml-v2-kserve-predictor-6578f8ffc7-cx7b9
19:44:02	kserve-ci-e2e-test	multus	isvc-pmml-v2-kserve-predictor-6578f8ffc7-cx7b9	AddedInterface	Add eth0 [10.133.0.24/23] from ovn-kubernetes
19:44:02	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-cx7b9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:44:02	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-cx7b9	Created	Created container: storage-initializer
19:44:03	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-2zztw	Unhealthy	Readiness probe failed: Get "https://10.133.0.23:8643/healthz": dial tcp 10.133.0.23:8643: connect: connection refused
19:44:03 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-runtime-predictor-67bc544947-2zztw	Unhealthy	Readiness probe failed: dial tcp 10.133.0.23:8080: connect: connection refused
19:44:07	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-cx7b9	Pulled	Container image "kserve/pmmlserver:latest" already present on machine
19:44:07	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-cx7b9	Created	Created container: kserve-container
19:44:07	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-cx7b9	Started	Started container kserve-container
19:44:07	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-cx7b9	Started	Started container kube-rbac-proxy
19:44:07	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-cx7b9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:44:07	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-cx7b9	Created	Created container: kube-rbac-proxy
19:44:10 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:44:16	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:44:16	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:45:24 (x10)	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-cx7b9	Unhealthy	Readiness probe failed: dial tcp 10.133.0.24:8080: connect: connection refused
19:45:27	kserve-ci-e2e-test	v1beta1Controllers	isvc-pmml-v2-kserve	InferenceServiceReady	InferenceService [isvc-pmml-v2-kserve] is Ready
19:45:31 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:45:31 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-pmml-v2-kserve-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:45:33	kserve-ci-e2e-test	replicaset-controller	isvc-primary-77d500-predictor-8d9ffc784	SuccessfulCreate	Created pod: isvc-primary-77d500-predictor-8d9ffc784-l4s82
19:45:33	kserve-ci-e2e-test	deployment-controller	isvc-primary-77d500-predictor	ScalingReplicaSet	Scaled up replica set isvc-primary-77d500-predictor-8d9ffc784 from 0 to 1
19:45:33	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-cx7b9	Killing	Stopping container kserve-container
19:45:33	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-cx7b9	Killing	Stopping container kube-rbac-proxy
19:45:34	kserve-ci-e2e-test	multus	isvc-primary-77d500-predictor-8d9ffc784-l4s82	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
19:45:34	kserve-ci-e2e-test	kubelet	isvc-primary-77d500-predictor-8d9ffc784-l4s82	Created	Created container: storage-initializer
19:45:34	kserve-ci-e2e-test	kubelet	isvc-primary-77d500-predictor-8d9ffc784-l4s82	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:45:34	kserve-ci-e2e-test	kubelet	isvc-primary-77d500-predictor-8d9ffc784-l4s82	Started	Started container storage-initializer
19:45:34	kserve-ci-e2e-test	kubelet	isvc-pmml-v2-kserve-predictor-6578f8ffc7-cx7b9	Unhealthy	Readiness probe failed: Get "https://10.133.0.24:8643/healthz": dial tcp 10.133.0.24:8643: connect: connection refused
19:45:38	kserve-ci-e2e-test	kubelet	isvc-primary-77d500-predictor-8d9ffc784-l4s82	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:45:38	kserve-ci-e2e-test	kubelet	isvc-primary-77d500-predictor-8d9ffc784-l4s82	Started	Started container kube-rbac-proxy
19:45:38	kserve-ci-e2e-test	kubelet	isvc-primary-77d500-predictor-8d9ffc784-l4s82	Created	Created container: kube-rbac-proxy
19:45:38	kserve-ci-e2e-test	kubelet	isvc-primary-77d500-predictor-8d9ffc784-l4s82	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1404" already present on machine
19:45:38	kserve-ci-e2e-test	kubelet	isvc-primary-77d500-predictor-8d9ffc784-l4s82	Created	Created container: kserve-container
19:45:38	kserve-ci-e2e-test	kubelet	isvc-primary-77d500-predictor-8d9ffc784-l4s82	Started	Started container kserve-container
19:46:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-77d500-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:46:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-77d500-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:46:36 (x9)	kserve-ci-e2e-test	kubelet	isvc-primary-77d500-predictor-8d9ffc784-l4s82	Unhealthy	Readiness probe failed: dial tcp 10.134.0.39:8080: connect: connection refused
19:46:46 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-77d500	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:46:46	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-77d500	InferenceServiceReady	InferenceService [isvc-primary-77d500] is Ready
19:46:52	kserve-ci-e2e-test	replicaset-controller	isvc-secondary-77d500-predictor-7cbd677c59	SuccessfulCreate	Created pod: isvc-secondary-77d500-predictor-7cbd677c59-wpc97
19:46:52	kserve-ci-e2e-test	deployment-controller	isvc-secondary-77d500-predictor	ScalingReplicaSet	Scaled up replica set isvc-secondary-77d500-predictor-7cbd677c59 from 0 to 1
19:46:52 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-77d500	UpdateFailed	Failed to update status for InferenceService "isvc-secondary-77d500": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-77d500": the object has been modified; please apply your changes to the latest version and try again
19:46:52 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-77d500	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-77d500": the object has been modified; please apply your changes to the latest version and try again
19:46:53	kserve-ci-e2e-test	multus	isvc-secondary-77d500-predictor-7cbd677c59-wpc97	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
19:46:57 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-77d500-predictor-7cbd677c59-wpc97	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:46:58 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-77d500-predictor-7cbd677c59-wpc97	Created	Created container: storage-initializer
19:46:58 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-77d500-predictor-7cbd677c59-wpc97	Started	Started container storage-initializer
19:47:02	kserve-ci-e2e-test	kubelet	isvc-secondary-77d500-predictor-7cbd677c59-wpc97	BackOff	Back-off restarting failed container storage-initializer in pod isvc-secondary-77d500-predictor-7cbd677c59-wpc97_kserve-ci-e2e-test(17d692a4-ac88-467f-97fe-4ec44b36dbb0)
19:47:03 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-77d500-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:47:03 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-77d500-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:47:04 (x14)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-77d500	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:47:07	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-77d500-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:47:07	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-77d500-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:47:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-19fe47	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-19fe47": the object has been modified; please apply your changes to the latest version and try again
19:47:08	kserve-ci-e2e-test	kubelet	isvc-primary-77d500-predictor-8d9ffc784-l4s82	Killing	Stopping container kserve-container
19:47:08	kserve-ci-e2e-test	kubelet	isvc-primary-77d500-predictor-8d9ffc784-l4s82	Killing	Stopping container kube-rbac-proxy
19:47:08	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-19fe47	UpdateFailed	Failed to update status for InferenceService "isvc-init-fail-19fe47": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-19fe47": the object has been modified; please apply your changes to the latest version and try again
19:47:08	kserve-ci-e2e-test	deployment-controller	isvc-init-fail-19fe47-predictor	ScalingReplicaSet	Scaled up replica set isvc-init-fail-19fe47-predictor-c5688fb5c from 0 to 1
19:47:09	kserve-ci-e2e-test	multus	isvc-init-fail-19fe47-predictor-c5688fb5c-bh8pm	AddedInterface	Add eth0 [10.134.0.41/23] from ovn-kubernetes
19:47:09	kserve-ci-e2e-test	replicaset-controller	isvc-init-fail-19fe47-predictor-c5688fb5c	SuccessfulCreate	Created pod: isvc-init-fail-19fe47-predictor-c5688fb5c-bh8pm
19:47:11	kserve-ci-e2e-test	kubelet	isvc-primary-77d500-predictor-8d9ffc784-l4s82	Unhealthy	Readiness probe failed: Get "https://10.134.0.39:8643/healthz": dial tcp 10.134.0.39:8643: connect: connection refused
19:47:14 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-19fe47-predictor-c5688fb5c-bh8pm	Started	Started container storage-initializer
19:47:14 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-19fe47-predictor-c5688fb5c-bh8pm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:47:14 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-19fe47-predictor-c5688fb5c-bh8pm	Created	Created container: storage-initializer
19:47:18	kserve-ci-e2e-test	kubelet	isvc-init-fail-19fe47-predictor-c5688fb5c-bh8pm	BackOff	Back-off restarting failed container storage-initializer in pod isvc-init-fail-19fe47-predictor-c5688fb5c-bh8pm_kserve-ci-e2e-test(42088543-26c0-4162-a75a-dfc5448d4af4)
19:47:18 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-19fe47	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:47:19	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-54b7w	Started	Started container storage-initializer
19:47:19	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-predictor-cd7c759c9-54b7w	AddedInterface	Add eth0 [10.133.0.25/23] from ovn-kubernetes
19:47:19	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-predictor-cd7c759c9 from 0 to 1
19:47:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-sklearn": the object has been modified; please apply your changes to the latest version and try again
19:47:19	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-54b7w	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:47:19	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-54b7w	Created	Created container: storage-initializer
19:47:19	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-predictor-cd7c759c9	SuccessfulCreate	Created pod: isvc-predictive-sklearn-predictor-cd7c759c9-54b7w
19:47:23	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-54b7w	Pulling	Pulling image "kserve/predictiveserver:latest"
19:47:26 (x24)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:47:47	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-54b7w	Created	Created container: kserve-container
19:47:47	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-54b7w	Started	Started container kserve-container
19:47:47	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-54b7w	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:47:47	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-54b7w	Created	Created container: kube-rbac-proxy
19:47:47	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-54b7w	Started	Started container kube-rbac-proxy
19:47:47	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-54b7w	Pulled	Successfully pulled image "kserve/predictiveserver:latest" in 23.906s (23.906s including waiting). Image size: 2325183853 bytes.
19:48:19 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:48:19 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:49:04 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:49:04 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:49:05	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn	InferenceServiceReady	InferenceService [isvc-predictive-sklearn] is Ready
19:49:13	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-54b7w	Killing	Stopping container kserve-container
19:49:13	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-54b7w	Killing	Stopping container kube-rbac-proxy
19:49:13	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-predictor-7ff98fd74d from 0 to 1
19:49:13	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost": the object has been modified; please apply your changes to the latest version and try again
19:49:14	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-predictor-7ff98fd74d-x6gsl	AddedInterface	Add eth0 [10.133.0.26/23] from ovn-kubernetes
19:49:14	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-x6gsl	Created	Created container: storage-initializer
19:49:14	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-54b7w	Unhealthy	Readiness probe failed: Get "https://10.133.0.25:8643/healthz": dial tcp 10.133.0.25:8643: connect: connection refused
19:49:14	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-x6gsl	Started	Started container storage-initializer
19:49:14 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-predictor-cd7c759c9-54b7w	Unhealthy	Readiness probe failed: dial tcp 10.133.0.25:8080: connect: connection refused
19:49:14	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-x6gsl	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:49:14	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-predictor-7ff98fd74d	SuccessfulCreate	Created pod: isvc-predictive-xgboost-predictor-7ff98fd74d-x6gsl
19:49:18	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-x6gsl	Started	Started container kserve-container
19:49:18	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-x6gsl	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
19:49:18	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-x6gsl	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:49:18	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-x6gsl	Created	Created container: kserve-container
19:49:19	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-x6gsl	Created	Created container: kube-rbac-proxy
19:49:19	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-x6gsl	Started	Started container kube-rbac-proxy
19:49:21 (x24)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:49:58 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:49:58 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:50:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost	InferenceServiceReady	InferenceService [isvc-predictive-xgboost] is Ready
19:50:43 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:50:43 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:50:47	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-predictor-75cb94f9f from 0 to 1
19:50:48	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-predictor-75cb94f9f	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-predictor-75cb94f9f-8whhc
19:50:48	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-predictor-75cb94f9f-8whhc	AddedInterface	Add eth0 [10.133.0.27/23] from ovn-kubernetes
19:50:48	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-8whhc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:50:48	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-8whhc	Created	Created container: storage-initializer
19:50:48	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-8whhc	Started	Started container storage-initializer
19:50:48	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-x6gsl	Killing	Stopping container kube-rbac-proxy
19:50:48	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-x6gsl	Killing	Stopping container kserve-container
19:50:50 (x11)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-x6gsl	Unhealthy	Readiness probe failed: dial tcp 10.133.0.26:8080: connect: connection refused
19:50:51	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-predictor-7ff98fd74d-x6gsl	Unhealthy	Readiness probe failed: Get "https://10.133.0.26:8643/healthz": dial tcp 10.133.0.26:8643: connect: connection refused
19:50:53	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-8whhc	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
19:50:53	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-8whhc	Started	Started container kube-rbac-proxy
19:50:53	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-8whhc	Created	Created container: kube-rbac-proxy
19:50:53	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-8whhc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:50:53	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-8whhc	Started	Started container kserve-container
19:50:53	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-8whhc	Created	Created container: kserve-container
19:50:56 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:51:32 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:51:32 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:52:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm] is Ready
19:52:18	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-8whhc	Killing	Stopping container kserve-container
19:52:18 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:52:18	kserve-ci-e2e-test	deployment-controller	isvc-predictive-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-sklearn-v2-predictor-b5d4f6b79 from 0 to 1
19:52:18	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-v2-predictor-b5d4f6b79	SuccessfulCreate	Created pod: isvc-predictive-sklearn-v2-predictor-b5d4f6b79-5l4f9
19:52:18	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-sklearn-v2-predictor-b5d4f6b79	FailedCreate	Error creating: Internal error occurred: failed calling webhook "inferenceservice.kserve-webhook-server.pod-mutator": failed to call webhook: Post "https://kserve-webhook-server-service.kserve.svc:443/mutate-pods?timeout=10s": EOF
19:52:18	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-8whhc	Killing	Stopping container kube-rbac-proxy
19:52:18 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:52:19	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-5l4f9	Started	Started container storage-initializer
19:52:19	kserve-ci-e2e-test	multus	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-5l4f9	AddedInterface	Add eth0 [10.133.0.28/23] from ovn-kubernetes
19:52:19	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-5l4f9	Created	Created container: storage-initializer
19:52:19	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-5l4f9	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:52:20	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-8whhc	Unhealthy	Readiness probe failed: Get "https://10.133.0.27:8643/healthz": dial tcp 10.133.0.27:8643: connect: connection refused
19:52:20 (x10)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-predictor-75cb94f9f-8whhc	Unhealthy	Readiness probe failed: dial tcp 10.133.0.27:8080: connect: connection refused
19:52:23	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-5l4f9	Started	Started container kube-rbac-proxy
19:52:23	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-5l4f9	Created	Created container: kserve-container
19:52:23	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-5l4f9	Started	Started container kserve-container
19:52:23	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-5l4f9	Created	Created container: kube-rbac-proxy
19:52:23	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-5l4f9	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
19:52:23	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-5l4f9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:52:26 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:53:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:53:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:53:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-sklearn-v2	InferenceServiceReady	InferenceService [isvc-predictive-sklearn-v2] is Ready
19:53:48	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-5l4f9	Killing	Stopping container kserve-container
19:53:48	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-predictive-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-predictive-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
19:53:48	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-5l4f9	Killing	Stopping container kube-rbac-proxy
19:53:48	kserve-ci-e2e-test	deployment-controller	isvc-predictive-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-xgboost-v2-predictor-6577c65fd8 from 0 to 1
19:53:48 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:53:48 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:53:48	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-xgboost-v2-predictor-6577c65fd8	SuccessfulCreate	Created pod: isvc-predictive-xgboost-v2-predictor-6577c65fd8-svdg6
19:53:49	kserve-ci-e2e-test	multus	isvc-predictive-xgboost-v2-predictor-6577c65fd8-svdg6	AddedInterface	Add eth0 [10.133.0.29/23] from ovn-kubernetes
19:53:49	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-svdg6	Created	Created container: storage-initializer
19:53:49	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-svdg6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:53:49	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-svdg6	Started	Started container storage-initializer
19:53:50	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-5l4f9	Unhealthy	Readiness probe failed: Get "https://10.133.0.28:8643/healthz": dial tcp 10.133.0.28:8643: connect: connection refused
19:53:50 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-sklearn-v2-predictor-b5d4f6b79-5l4f9	Unhealthy	Readiness probe failed: Get "http://10.133.0.28:8080/v2/models/isvc-predictive-sklearn-v2/ready": dial tcp 10.133.0.28:8080: connect: connection refused
19:53:53	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-svdg6	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:53:53	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-svdg6	Created	Created container: kserve-container
19:53:53	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-svdg6	Started	Started container kserve-container
19:53:53	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-svdg6	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
19:53:53	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-svdg6	Created	Created container: kube-rbac-proxy
19:53:53	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-svdg6	Started	Started container kube-rbac-proxy
19:53:56 (x24)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:54:33 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:54:33 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:55:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-xgboost-v2	InferenceServiceReady	InferenceService [isvc-predictive-xgboost-v2] is Ready
19:55:18	kserve-ci-e2e-test	deployment-controller	isvc-predictive-lightgbm-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-predictive-lightgbm-v2-predictor-865b4598f7 from 0 to 1
19:55:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:55:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:55:19	kserve-ci-e2e-test	multus	isvc-predictive-lightgbm-v2-predictor-865b4598f7-ftgmz	AddedInterface	Add eth0 [10.133.0.30/23] from ovn-kubernetes
19:55:19	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-ftgmz	Started	Started container storage-initializer
19:55:19	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-ftgmz	Created	Created container: storage-initializer
19:55:19	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-ftgmz	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:55:19	kserve-ci-e2e-test	replicaset-controller	isvc-predictive-lightgbm-v2-predictor-865b4598f7	SuccessfulCreate	Created pod: isvc-predictive-lightgbm-v2-predictor-865b4598f7-ftgmz
19:55:19	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-svdg6	Killing	Stopping container kube-rbac-proxy
19:55:19	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-svdg6	Killing	Stopping container kserve-container
19:55:20	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-svdg6	Unhealthy	Readiness probe failed: Get "https://10.133.0.29:8643/healthz": dial tcp 10.133.0.29:8643: connect: connection refused
19:55:20 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-xgboost-v2-predictor-6577c65fd8-svdg6	Unhealthy	Readiness probe failed: Get "http://10.133.0.29:8080/v2/models/isvc-predictive-xgboost-v2/ready": dial tcp 10.133.0.29:8080: connect: connection refused
19:55:24	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-ftgmz	Pulled	Container image "kserve/predictiveserver:latest" already present on machine
19:55:24	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-ftgmz	Started	Started container kube-rbac-proxy
19:55:24	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-ftgmz	Created	Created container: kube-rbac-proxy
19:55:24	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-ftgmz	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:55:24	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-ftgmz	Started	Started container kserve-container
19:55:24	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-ftgmz	Created	Created container: kserve-container
19:55:27 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:56:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:56:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:56:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-predictive-lightgbm-v2	InferenceServiceReady	InferenceService [isvc-predictive-lightgbm-v2] is Ready
19:56:48 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:56:48	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-scheduler-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-scheduler-predictor-cd9c59d7 from 0 to 1
19:56:48 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-predictive-lightgbm-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:56:49	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-ftgmz	Killing	Stopping container kube-rbac-proxy
19:56:49	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-scheduler-predictor-cd9c59d7	SuccessfulCreate	Created pod: isvc-sklearn-scheduler-predictor-cd9c59d7-9t72f
19:56:49 (x3)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scheduler	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:56:49	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-ftgmz	Killing	Stopping container kserve-container
19:56:50	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn": the object has been modified; please apply your changes to the latest version and try again
19:56:50	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-predictor-6875c879b7	SuccessfulCreate	Created pod: isvc-sklearn-predictor-6875c879b7-96mbk
19:56:50	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-predictor-6875c879b7 from 0 to 1
19:56:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6875c879b7-96mbk	Created	Created container: storage-initializer
19:56:51	kserve-ci-e2e-test	multus	isvc-sklearn-predictor-6875c879b7-96mbk	AddedInterface	Add eth0 [10.134.0.42/23] from ovn-kubernetes
19:56:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6875c879b7-96mbk	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:56:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6875c879b7-96mbk	Started	Started container storage-initializer
19:56:52 (x5)	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-ftgmz	Unhealthy	Readiness probe failed: Get "http://10.133.0.30:8080/v2/models/isvc-predictive-lightgbm-v2/ready": dial tcp 10.133.0.30:8080: connect: connection refused
19:56:52	kserve-ci-e2e-test	kubelet	isvc-predictive-lightgbm-v2-predictor-865b4598f7-ftgmz	Unhealthy	Readiness probe failed: Get "https://10.133.0.30:8643/healthz": dial tcp 10.133.0.30:8643: connect: connection refused
19:56:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6875c879b7-96mbk	Created	Created container: kube-rbac-proxy
19:56:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6875c879b7-96mbk	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:56:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6875c879b7-96mbk	Started	Started container kserve-container
19:56:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6875c879b7-96mbk	Created	Created container: kserve-container
19:56:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6875c879b7-96mbk	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1404" already present on machine
19:56:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6875c879b7-96mbk	Started	Started container kube-rbac-proxy
19:56:58 (x24)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:57:05	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:57:05	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:58:02	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn	InferenceServiceReady	InferenceService [isvc-sklearn] is Ready
19:58:05 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:58:05 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:58:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6875c879b7-96mbk	Killing	Stopping container kube-rbac-proxy
19:58:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6875c879b7-96mbk	Killing	Stopping container kserve-container
19:58:11	kserve-ci-e2e-test	replicaset-controller	sklearn-v2-mlserver-predictor-65d8664766	SuccessfulCreate	Created pod: sklearn-v2-mlserver-predictor-65d8664766-mw5mr
19:58:11	kserve-ci-e2e-test	deployment-controller	sklearn-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set sklearn-v2-mlserver-predictor-65d8664766 from 0 to 1
19:58:12	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-mw5mr	Created	Created container: storage-initializer
19:58:12	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-mw5mr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:58:12	kserve-ci-e2e-test	multus	sklearn-v2-mlserver-predictor-65d8664766-mw5mr	AddedInterface	Add eth0 [10.134.0.43/23] from ovn-kubernetes
19:58:12 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6875c879b7-96mbk	Unhealthy	Readiness probe failed: dial tcp 10.134.0.42:8080: connect: connection refused
19:58:12	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-mw5mr	Started	Started container storage-initializer
19:58:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-predictor-6875c879b7-96mbk	Unhealthy	Readiness probe failed: Get "https://10.134.0.42:8643/healthz": dial tcp 10.134.0.42:8643: connect: connection refused
19:58:16	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-mw5mr	Created	Created container: kserve-container
19:58:16	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-mw5mr	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:58:16	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-mw5mr	Started	Started container kserve-container
19:58:16	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-mw5mr	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:58:16	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-mw5mr	Created	Created container: kube-rbac-proxy
19:58:16	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-mw5mr	Started	Started container kube-rbac-proxy
19:58:19 (x25)	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:58:41 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:58:41 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:58:54	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-mw5mr	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
19:59:04	kserve-ci-e2e-test	v1beta1Controllers	sklearn-v2-mlserver	InferenceServiceReady	InferenceService [sklearn-v2-mlserver] is Ready
19:59:11	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-mw5mr	Killing	Stopping container kube-rbac-proxy
19:59:11	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-mw5mr	Killing	Stopping container kserve-container
19:59:11	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-runtime-predictor-7c6499f57	SuccessfulCreate	Created pod: isvc-sklearn-runtime-predictor-7c6499f57-hk2ls
19:59:11 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:59:11 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sklearn-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
19:59:11	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-runtime-predictor-7c6499f57 from 0 to 1
19:59:12	kserve-ci-e2e-test	multus	isvc-sklearn-runtime-predictor-7c6499f57-hk2ls	AddedInterface	Add eth0 [10.134.0.44/23] from ovn-kubernetes
19:59:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-7c6499f57-hk2ls	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:59:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-7c6499f57-hk2ls	Created	Created container: storage-initializer
19:59:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-7c6499f57-hk2ls	Started	Started container storage-initializer
19:59:14	kserve-ci-e2e-test	kubelet	sklearn-v2-mlserver-predictor-65d8664766-mw5mr	Unhealthy	Readiness probe failed: Get "https://10.134.0.43:8643/healthz": dial tcp 10.134.0.43:8643: connect: connection refused
19:59:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-7c6499f57-hk2ls	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1404" already present on machine
19:59:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-7c6499f57-hk2ls	Created	Created container: kube-rbac-proxy
19:59:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-7c6499f57-hk2ls	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:59:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-7c6499f57-hk2ls	Started	Started container kube-rbac-proxy
19:59:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-7c6499f57-hk2ls	Created	Created container: kserve-container
19:59:18	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-7c6499f57-hk2ls	Started	Started container kserve-container
19:59:19 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
19:59:25 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-7c6499f57-hk2ls	Unhealthy	Readiness probe failed: dial tcp 10.134.0.44:8080: connect: connection refused
19:59:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-runtime] is Ready
19:59:41 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:59:41 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
19:59:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-7c6499f57-hk2ls	Killing	Stopping container kube-rbac-proxy
19:59:48	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-runtime-predictor-6d84c876f4 from 0 to 1
19:59:48	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-runtime-predictor-6d84c876f4	SuccessfulCreate	Created pod: isvc-sklearn-v2-runtime-predictor-6d84c876f4-8ctdr
19:59:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-runtime-predictor-7c6499f57-hk2ls	Killing	Stopping container kserve-container
19:59:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8ctdr	Started	Started container storage-initializer
19:59:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8ctdr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
19:59:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8ctdr	Created	Created container: storage-initializer
19:59:49	kserve-ci-e2e-test	multus	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8ctdr	AddedInterface	Add eth0 [10.134.0.45/23] from ovn-kubernetes
19:59:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8ctdr	Created	Created container: kserve-container
19:59:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8ctdr	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
19:59:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8ctdr	Started	Started container kserve-container
19:59:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8ctdr	Started	Started container kube-rbac-proxy
19:59:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8ctdr	Created	Created container: kube-rbac-proxy
19:59:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8ctdr	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
19:59:57 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:00:03	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:00:03	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:00:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8ctdr	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
20:00:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-runtime	InferenceServiceReady	InferenceService [isvc-sklearn-v2-runtime] is Ready
20:00:48	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-predictor-64fcb8589f from 0 to 1
20:00:48 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:00:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8ctdr	Killing	Stopping container kube-rbac-proxy
20:00:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8ctdr	Killing	Stopping container kserve-container
20:00:48 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:00:49	kserve-ci-e2e-test	multus	isvc-sklearn-v2-predictor-64fcb8589f-wgmk5	AddedInterface	Add eth0 [10.134.0.46/23] from ovn-kubernetes
20:00:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-64fcb8589f-wgmk5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
20:00:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-64fcb8589f-wgmk5	Created	Created container: storage-initializer
20:00:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-64fcb8589f-wgmk5	Started	Started container storage-initializer
20:00:49	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-predictor-64fcb8589f	SuccessfulCreate	Created pod: isvc-sklearn-v2-predictor-64fcb8589f-wgmk5
20:00:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8ctdr	Unhealthy	Readiness probe failed: Get "http://10.134.0.45:8080/v2/models/isvc-sklearn-v2-runtime/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:00:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-64fcb8589f-wgmk5	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:00:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-64fcb8589f-wgmk5	Created	Created container: kube-rbac-proxy
20:00:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-64fcb8589f-wgmk5	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1404" already present on machine
20:00:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-64fcb8589f-wgmk5	Created	Created container: kserve-container
20:00:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-64fcb8589f-wgmk5	Started	Started container kserve-container
20:00:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-64fcb8589f-wgmk5	Started	Started container kube-rbac-proxy
20:00:56 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-runtime-predictor-6d84c876f4-8ctdr	Unhealthy	Readiness probe failed: Get "https://10.134.0.45:8643/healthz": dial tcp 10.134.0.45:8643: connect: connection refused
20:00:57 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:01:03	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:01:03	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:02:00	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2	InferenceServiceReady	InferenceService [isvc-sklearn-v2] is Ready
20:02:03 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:02:03 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:02:08	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-v2-mixed-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-v2-mixed-predictor-86d7579fd6 from 0 to 1
20:02:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-86d7579fd6-87x9p	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
20:02:09	kserve-ci-e2e-test	multus	isvc-sklearn-v2-mixed-predictor-86d7579fd6-87x9p	AddedInterface	Add eth0 [10.134.0.47/23] from ovn-kubernetes
20:02:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-64fcb8589f-wgmk5	Killing	Stopping container kserve-container
20:02:09	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-v2-mixed-predictor-86d7579fd6	SuccessfulCreate	Created pod: isvc-sklearn-v2-mixed-predictor-86d7579fd6-87x9p
20:02:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-64fcb8589f-wgmk5	Killing	Stopping container kube-rbac-proxy
20:02:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-86d7579fd6-87x9p	Started	Started container storage-initializer
20:02:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-86d7579fd6-87x9p	Created	Created container: storage-initializer
20:02:10 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-64fcb8589f-wgmk5	Unhealthy	Readiness probe failed: dial tcp 10.134.0.46:8080: connect: connection refused
20:02:10	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-predictor-64fcb8589f-wgmk5	Unhealthy	Readiness probe failed: Get "https://10.134.0.46:8643/healthz": dial tcp 10.134.0.46:8643: connect: connection refused
20:02:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-86d7579fd6-87x9p	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1404" already present on machine
20:02:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-86d7579fd6-87x9p	Started	Started container kserve-container
20:02:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-86d7579fd6-87x9p	Created	Created container: kserve-container
20:02:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-86d7579fd6-87x9p	Started	Started container kube-rbac-proxy
20:02:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-86d7579fd6-87x9p	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:02:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-86d7579fd6-87x9p	Created	Created container: kube-rbac-proxy
20:02:17 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:02:23	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:02:23	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:03:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-v2-mixed	InferenceServiceReady	InferenceService [isvc-sklearn-v2-mixed] is Ready
20:03:23 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:03:23 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-v2-mixed-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:03:29	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-predictor-6756f669d7	SuccessfulCreate	Created pod: isvc-tensorflow-predictor-6756f669d7-wpjsw
20:03:29	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-predictor-6756f669d7 from 0 to 1
20:03:29	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-wpjsw	Created	Created container: storage-initializer
20:03:29	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-wpjsw	Started	Started container storage-initializer
20:03:29	kserve-ci-e2e-test	multus	isvc-tensorflow-predictor-6756f669d7-wpjsw	AddedInterface	Add eth0 [10.133.0.31/23] from ovn-kubernetes
20:03:29	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-wpjsw	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
20:03:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-86d7579fd6-87x9p	Killing	Stopping container kube-rbac-proxy
20:03:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-86d7579fd6-87x9p	Killing	Stopping container kserve-container
20:03:30	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-86d7579fd6-87x9p	Unhealthy	Readiness probe failed: Get "https://10.134.0.47:8643/healthz": dial tcp 10.134.0.47:8643: connect: connection refused
20:03:30 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-v2-mixed-predictor-86d7579fd6-87x9p	Unhealthy	Readiness probe failed: dial tcp 10.134.0.47:8080: connect: connection refused
20:03:34	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-wpjsw	Pulling	Pulling image "tensorflow/serving:2.6.2"
20:03:37 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:03:38	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-wpjsw	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:03:38	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-wpjsw	Created	Created container: kube-rbac-proxy
20:03:38	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-wpjsw	Started	Started container kube-rbac-proxy
20:03:38	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-wpjsw	Started	Started container kserve-container
20:03:38	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-wpjsw	Pulled	Successfully pulled image "tensorflow/serving:2.6.2" in 4.455s (4.455s including waiting). Image size: 425873876 bytes.
20:03:38	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-wpjsw	Created	Created container: kserve-container
20:03:46 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-wpjsw	Unhealthy	Readiness probe failed: dial tcp 10.133.0.31:8080: connect: connection refused
20:03:56	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow	InferenceServiceReady	InferenceService [isvc-tensorflow] is Ready
20:04:14 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:04:14	kserve-ci-e2e-test	replicaset-controller	isvc-tensorflow-runtime-predictor-8699d78cf	SuccessfulCreate	Created pod: isvc-tensorflow-runtime-predictor-8699d78cf-fh7l5
20:04:14	kserve-ci-e2e-test	deployment-controller	isvc-tensorflow-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-tensorflow-runtime-predictor-8699d78cf from 0 to 1
20:04:14 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:04:14	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-wpjsw	Killing	Stopping container kube-rbac-proxy
20:04:14	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-wpjsw	Killing	Stopping container kserve-container
20:04:15	kserve-ci-e2e-test	multus	isvc-tensorflow-runtime-predictor-8699d78cf-fh7l5	AddedInterface	Add eth0 [10.133.0.32/23] from ovn-kubernetes
20:04:15	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-fh7l5	Started	Started container storage-initializer
20:04:15	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-fh7l5	Created	Created container: storage-initializer
20:04:15	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-fh7l5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
20:04:20	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-fh7l5	Pulled	Container image "tensorflow/serving:2.6.2" already present on machine
20:04:20	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-fh7l5	Created	Created container: kserve-container
20:04:20	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-fh7l5	Started	Started container kserve-container
20:04:20	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-fh7l5	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:04:20	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-fh7l5	Created	Created container: kube-rbac-proxy
20:04:20	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-fh7l5	Started	Started container kube-rbac-proxy
20:04:23 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:04:28 (x3)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-fh7l5	Unhealthy	Readiness probe failed: dial tcp 10.133.0.32:8080: connect: connection refused
20:04:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-tensorflow-runtime	InferenceServiceReady	InferenceService [isvc-tensorflow-runtime] is Ready
20:04:41 (x6)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-predictor-6756f669d7-wpjsw	Unhealthy	Readiness probe failed: Get "https://10.133.0.31:8643/healthz": dial tcp 10.133.0.31:8643: connect: connection refused
20:04:44 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:04:44 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-tensorflow-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:04:56	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-fh7l5	Killing	Stopping container kube-rbac-proxy
20:04:56	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-jvtld	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
20:04:56	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-jvtld	Created	Created container: storage-initializer
20:04:56	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-jvtld	Started	Started container storage-initializer
20:04:56	kserve-ci-e2e-test	multus	isvc-triton-predictor-84bb65d94b-jvtld	AddedInterface	Add eth0 [10.133.0.33/23] from ovn-kubernetes
20:04:56	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-fh7l5	Killing	Stopping container kserve-container
20:04:56	kserve-ci-e2e-test	replicaset-controller	isvc-triton-predictor-84bb65d94b	SuccessfulCreate	Created pod: isvc-triton-predictor-84bb65d94b-jvtld
20:04:56	kserve-ci-e2e-test	deployment-controller	isvc-triton-predictor	ScalingReplicaSet	Scaled up replica set isvc-triton-predictor-84bb65d94b from 0 to 1
20:05:01	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-jvtld	Pulling	Pulling image "nvcr.io/nvidia/tritonserver:23.05-py3"
20:05:04 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:05:23 (x6)	kserve-ci-e2e-test	kubelet	isvc-tensorflow-runtime-predictor-8699d78cf-fh7l5	Unhealthy	Readiness probe failed: Get "https://10.133.0.32:8643/healthz": dial tcp 10.133.0.32:8643: connect: connection refused
20:06:56	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-jvtld	Created	Created container: kserve-container
20:06:56	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-jvtld	Pulled	Successfully pulled image "nvcr.io/nvidia/tritonserver:23.05-py3" in 1m55.146s (1m55.146s including waiting). Image size: 12907074623 bytes.
20:06:56	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-jvtld	Created	Created container: kube-rbac-proxy
20:06:56 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:06:56 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-triton-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:06:56	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-jvtld	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:06:56	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-jvtld	Started	Started container kserve-container
20:06:57	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-jvtld	Started	Started container kube-rbac-proxy
20:06:59 (x2)	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-jvtld	Unhealthy	Readiness probe failed: dial tcp 10.133.0.33:8080: connect: connection refused
20:07:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-triton	InferenceServiceReady	InferenceService [isvc-triton] is Ready
20:07:07	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-jvtld	Killing	Stopping container kserve-container
20:07:07	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-jvtld	Killing	Stopping container kube-rbac-proxy
20:07:07	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-predictor-8689c4cfcc from 0 to 1
20:07:08	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-predictor-8689c4cfcc	SuccessfulCreate	Created pod: isvc-xgboost-predictor-8689c4cfcc-w8f78
20:07:08	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-w8f78	Created	Created container: storage-initializer
20:07:08	kserve-ci-e2e-test	multus	isvc-xgboost-predictor-8689c4cfcc-w8f78	AddedInterface	Add eth0 [10.133.0.34/23] from ovn-kubernetes
20:07:08	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-w8f78	Started	Started container storage-initializer
20:07:08	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-w8f78	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
20:07:09	kserve-ci-e2e-test	kubelet	isvc-triton-predictor-84bb65d94b-jvtld	Unhealthy	Readiness probe failed: Get "https://10.133.0.33:8643/healthz": dial tcp 10.133.0.33:8643: connect: connection refused
20:07:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-w8f78	Pulling	Pulling image "kserve/xgbserver:latest"
20:07:16 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:07:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-w8f78	Started	Started container kserve-container
20:07:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-w8f78	Started	Started container kube-rbac-proxy
20:07:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-w8f78	Created	Created container: kube-rbac-proxy
20:07:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-w8f78	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:07:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-w8f78	Created	Created container: kserve-container
20:07:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-w8f78	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 21.566s (21.566s including waiting). Image size: 1306329851 bytes.
20:08:07 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:08:07 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:08:37 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:08:37 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:08:41	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost	InferenceServiceReady	InferenceService [isvc-xgboost] is Ready
20:08:47	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-w8f78	Killing	Stopping container kube-rbac-proxy
20:08:47	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-mlserver-predictor-67d4bc6646 from 0 to 1
20:08:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
20:08:47	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-w8f78	Killing	Stopping container kserve-container
20:08:48	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-cq8m2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
20:08:48	kserve-ci-e2e-test	multus	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-cq8m2	AddedInterface	Add eth0 [10.134.0.48/23] from ovn-kubernetes
20:08:48	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-cq8m2	Created	Created container: storage-initializer
20:08:48	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-cq8m2	Started	Started container storage-initializer
20:08:48	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-mlserver-predictor-67d4bc6646	SuccessfulCreate	Created pod: isvc-xgboost-v2-mlserver-predictor-67d4bc6646-cq8m2
20:08:51	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-w8f78	Unhealthy	Readiness probe failed: Get "https://10.133.0.34:8643/healthz": dial tcp 10.133.0.34:8643: connect: connection refused
20:08:51 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-predictor-8689c4cfcc-w8f78	Unhealthy	Readiness probe failed: dial tcp 10.133.0.34:8080: connect: connection refused
20:08:52	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-cq8m2	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
20:08:53	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-cq8m2	Created	Created container: kube-rbac-proxy
20:08:53	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-cq8m2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:08:53	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-cq8m2	Started	Started container kube-rbac-proxy
20:08:53	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-cq8m2	Started	Started container kserve-container
20:08:53	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-cq8m2	Created	Created container: kserve-container
20:08:56 (x24)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:09:17	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:09:17	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:09:30	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-mlserver	InferenceServiceReady	InferenceService [isvc-xgboost-v2-mlserver] is Ready
20:09:33 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:09:33 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:09:38	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-cq8m2	Killing	Stopping container kserve-container
20:09:38	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-cq8m2	Killing	Stopping container kube-rbac-proxy
20:09:38	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	UpdateFailed	Failed to update status for InferenceService "xgboost-v2-mlserver": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "xgboost-v2-mlserver": the object has been modified; please apply your changes to the latest version and try again
20:09:38	kserve-ci-e2e-test	deployment-controller	xgboost-v2-mlserver-predictor	ScalingReplicaSet	Scaled up replica set xgboost-v2-mlserver-predictor-7799869d6f from 0 to 1
20:09:38	kserve-ci-e2e-test	replicaset-controller	xgboost-v2-mlserver-predictor-7799869d6f	SuccessfulCreate	Created pod: xgboost-v2-mlserver-predictor-7799869d6f-v4hvs
20:09:39	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-v4hvs	Started	Started container storage-initializer
20:09:39	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-v4hvs	Created	Created container: storage-initializer
20:09:39	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-v4hvs	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
20:09:39	kserve-ci-e2e-test	multus	xgboost-v2-mlserver-predictor-7799869d6f-v4hvs	AddedInterface	Add eth0 [10.134.0.49/23] from ovn-kubernetes
20:09:43	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-v4hvs	Started	Started container kserve-container
20:09:43	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-v4hvs	Created	Created container: kube-rbac-proxy
20:09:43	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-v4hvs	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:09:43	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-v4hvs	Created	Created container: kserve-container
20:09:43	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-v4hvs	Started	Started container kube-rbac-proxy
20:09:43	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-v4hvs	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
20:09:44 (x2)	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-mlserver-predictor-67d4bc6646-cq8m2	Unhealthy	Readiness probe failed: Get "https://10.134.0.48:8643/healthz": dial tcp 10.134.0.48:8643: connect: connection refused
20:09:46 (x24)	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:10:20	kserve-ci-e2e-test	v1beta1Controllers	xgboost-v2-mlserver	InferenceServiceReady	InferenceService [xgboost-v2-mlserver] is Ready
20:10:23 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:10:23 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	xgboost-v2-mlserver-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:10:28	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-v4hvs	Killing	Stopping container kserve-container
20:10:28	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-v4hvs	Killing	Stopping container kube-rbac-proxy
20:10:28	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-runtime-predictor-779db84d9	SuccessfulCreate	Created pod: isvc-xgboost-runtime-predictor-779db84d9-w5ldr
20:10:28	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-runtime-predictor-779db84d9 from 0 to 1
20:10:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-w5ldr	Created	Created container: storage-initializer
20:10:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-w5ldr	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
20:10:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-w5ldr	Started	Started container storage-initializer
20:10:29	kserve-ci-e2e-test	multus	isvc-xgboost-runtime-predictor-779db84d9-w5ldr	AddedInterface	Add eth0 [10.133.0.35/23] from ovn-kubernetes
20:10:33	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-w5ldr	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:10:33	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-w5ldr	Created	Created container: kserve-container
20:10:33	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-w5ldr	Started	Started container kserve-container
20:10:33	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-w5ldr	Created	Created container: kube-rbac-proxy
20:10:33	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-w5ldr	Started	Started container kube-rbac-proxy
20:10:33	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-w5ldr	Pulled	Container image "kserve/xgbserver:latest" already present on machine
20:10:35 (x2)	kserve-ci-e2e-test	kubelet	xgboost-v2-mlserver-predictor-7799869d6f-v4hvs	Unhealthy	Readiness probe failed: Get "https://10.134.0.49:8643/healthz": dial tcp 10.134.0.49:8643: connect: connection refused
20:10:36 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:11:13 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:11:13 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:11:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-runtime] is Ready
20:11:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:11:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:11:48	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-w5ldr	Killing	Stopping container kube-rbac-proxy
20:11:48	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-runtime-predictor-6dc5954dc	SuccessfulCreate	Created pod: isvc-xgboost-v2-runtime-predictor-6dc5954dc-sp5z2
20:11:48	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-runtime-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-runtime-predictor-6dc5954dc from 0 to 1
20:11:48	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-w5ldr	Killing	Stopping container kserve-container
20:11:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-sp5z2	Created	Created container: storage-initializer
20:11:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-sp5z2	Started	Started container storage-initializer
20:11:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-sp5z2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
20:11:49	kserve-ci-e2e-test	multus	isvc-xgboost-v2-runtime-predictor-6dc5954dc-sp5z2	AddedInterface	Add eth0 [10.134.0.50/23] from ovn-kubernetes
20:11:50	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-w5ldr	Unhealthy	Readiness probe failed: Get "https://10.133.0.35:8643/healthz": dial tcp 10.133.0.35:8643: connect: connection refused
20:11:50 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-runtime-predictor-779db84d9-w5ldr	Unhealthy	Readiness probe failed: dial tcp 10.133.0.35:8080: connect: connection refused
20:11:53	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-sp5z2	Created	Created container: kserve-container
20:11:53	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-sp5z2	Started	Started container kserve-container
20:11:53	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-sp5z2	Created	Created container: kube-rbac-proxy
20:11:53	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-sp5z2	Pulled	Container image "docker.io/seldonio/mlserver:1.7.1" already present on machine
20:11:53	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-sp5z2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:11:53	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-sp5z2	Started	Started container kube-rbac-proxy
20:11:57 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:12:30	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-sp5z2	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 400
20:12:33 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:12:33 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:12:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2-runtime	InferenceServiceReady	InferenceService [isvc-xgboost-v2-runtime] is Ready
20:12:48	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-sp5z2	Killing	Stopping container kserve-container
20:12:48	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-v2-predictor-6fcdd6977c from 0 to 1
20:12:48	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-sp5z2	Killing	Stopping container kube-rbac-proxy
20:12:48	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:12:48	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-runtime-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:12:48	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-v2": the object has been modified; please apply your changes to the latest version and try again
20:12:49	kserve-ci-e2e-test	multus	isvc-xgboost-v2-predictor-6fcdd6977c-k9kkk	AddedInterface	Add eth0 [10.133.0.36/23] from ovn-kubernetes
20:12:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-k9kkk	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
20:12:49	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-v2-predictor-6fcdd6977c	SuccessfulCreate	Created pod: isvc-xgboost-v2-predictor-6fcdd6977c-k9kkk
20:12:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-k9kkk	Started	Started container storage-initializer
20:12:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-k9kkk	Created	Created container: storage-initializer
20:12:51	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-sp5z2	Unhealthy	Readiness probe failed: Get "http://10.134.0.50:8080/v2/models/isvc-xgboost-v2-runtime/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
20:12:54	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-k9kkk	Created	Created container: kube-rbac-proxy
20:12:54	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-k9kkk	Created	Created container: kserve-container
20:12:54	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-k9kkk	Started	Started container kube-rbac-proxy
20:12:54	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-k9kkk	Pulled	Container image "kserve/xgbserver:latest" already present on machine
20:12:54	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-k9kkk	Started	Started container kserve-container
20:12:54	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-k9kkk	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:12:55 (x2)	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-runtime-predictor-6dc5954dc-sp5z2	Unhealthy	Readiness probe failed: Get "https://10.134.0.50:8643/healthz": dial tcp 10.134.0.50:8643: connect: connection refused
20:12:57 (x24)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:13:03	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:13:03	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:14:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-v2	InferenceServiceReady	InferenceService [isvc-xgboost-v2] is Ready
20:14:03 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:14:03 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:14:08	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-predictor-584b446894 from 0 to 1
20:14:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-584b446894-dhtlj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
20:14:09	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-k9kkk	Killing	Stopping container kserve-container
20:14:09	kserve-ci-e2e-test	multus	isvc-sklearn-s3-predictor-584b446894-dhtlj	AddedInterface	Add eth0 [10.134.0.51/23] from ovn-kubernetes
20:14:09	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-predictor-584b446894	SuccessfulCreate	Created pod: isvc-sklearn-s3-predictor-584b446894-dhtlj
20:14:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-584b446894-dhtlj	Created	Created container: storage-initializer
20:14:09	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-k9kkk	Killing	Stopping container kube-rbac-proxy
20:14:09	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-584b446894-dhtlj	Started	Started container storage-initializer
20:14:10	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-584b446894-dhtlj	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1404" already present on machine
20:14:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-584b446894-dhtlj	Created	Created container: kserve-container
20:14:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-584b446894-dhtlj	Created	Created container: kube-rbac-proxy
20:14:11	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-k9kkk	Unhealthy	Readiness probe failed: Get "https://10.133.0.36:8643/healthz": dial tcp 10.133.0.36:8643: connect: connection refused
20:14:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-584b446894-dhtlj	Started	Started container kserve-container
20:14:11 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-v2-predictor-6fcdd6977c-k9kkk	Unhealthy	Readiness probe failed: dial tcp 10.133.0.36:8080: connect: connection refused
20:14:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-584b446894-dhtlj	Started	Started container kube-rbac-proxy
20:14:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-584b446894-dhtlj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:14:17 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:14:38 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:14:38 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:15:19	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3	InferenceServiceReady	InferenceService [isvc-sklearn-s3] is Ready
20:15:23 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:15:23 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:15:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-584b446894-dhtlj	Unhealthy	Readiness probe failed: Get "https://10.134.0.51:8643/healthz": dial tcp 10.134.0.51:8643: connect: connection refused
20:15:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-584b446894-dhtlj	Killing	Stopping container kserve-container
20:15:28	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-584b446894-dhtlj	Killing	Stopping container kube-rbac-proxy
20:15:28 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-predictor-584b446894-dhtlj	Unhealthy	Readiness probe failed: dial tcp 10.134.0.51:8080: connect: connection refused
20:15:29	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-pass-predictor-764bc9bb86 from 0 to 1
20:15:29	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-pass-predictor-764bc9bb86-5qmr4	AddedInterface	Add eth0 [10.134.0.52/23] from ovn-kubernetes
20:15:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-764bc9bb86-5qmr4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
20:15:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-764bc9bb86-5qmr4	Created	Created container: storage-initializer
20:15:29	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-764bc9bb86-5qmr4	Started	Started container storage-initializer
20:15:29	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-pass-predictor-764bc9bb86	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-pass-predictor-764bc9bb86-5qmr4
20:15:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-764bc9bb86-5qmr4	Started	Started container kserve-container
20:15:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-764bc9bb86-5qmr4	Started	Started container kube-rbac-proxy
20:15:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-764bc9bb86-5qmr4	Created	Created container: kube-rbac-proxy
20:15:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-764bc9bb86-5qmr4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:15:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-764bc9bb86-5qmr4	Created	Created container: kserve-container
20:15:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-764bc9bb86-5qmr4	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1404" already present on machine
20:15:37 (x25)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:15:59 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:15:59 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:16:28 (x8)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-764bc9bb86-5qmr4	Unhealthy	Readiness probe failed: dial tcp 10.134.0.52:8080: connect: connection refused
20:16:29 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:16:29 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-global-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:16:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-global-pass] is Ready
20:16:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-global-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-global-fail": the object has been modified; please apply your changes to the latest version and try again
20:16:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-764bc9bb86-5qmr4	Killing	Stopping container kserve-container
20:16:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-764bc9bb86-5qmr4	Killing	Stopping container kube-rbac-proxy
20:16:39	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-global-fail-predictor-8cfdd8b8d	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-global-fail-predictor-8cfdd8b8d-nx7ll
20:16:39	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-global-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-global-fail-predictor-8cfdd8b8d from 0 to 1
20:16:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-8cfdd8b8d-nx7ll	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
20:16:40	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-global-fail-predictor-8cfdd8b8d-nx7ll	AddedInterface	Add eth0 [10.134.0.53/23] from ovn-kubernetes
20:16:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-pass-predictor-764bc9bb86-5qmr4	Unhealthy	Readiness probe failed: Get "https://10.134.0.52:8643/healthz": dial tcp 10.134.0.52:8643: connect: connection refused
20:16:45 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-8cfdd8b8d-nx7ll	Started	Started container storage-initializer
20:16:45 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-8cfdd8b8d-nx7ll	Created	Created container: storage-initializer
20:16:45 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-8cfdd8b8d-nx7ll	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
20:16:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-global-fail-predictor-8cfdd8b8d-nx7ll	BackOff	Back-off restarting failed container storage-initializer in pod isvc-sklearn-s3-tls-global-fail-predictor-8cfdd8b8d-nx7ll_kserve-ci-e2e-test(0cec68ac-098a-4035-973d-4ae9e7f46cce)
20:16:49 (x23)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-global-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:16:50	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-pass-predictor-848ff9798 from 0 to 1
20:16:51	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-pass": the object has been modified; please apply your changes to the latest version and try again
20:16:51 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-pass": the object has been modified; please apply your changes to the latest version and try again
20:16:51	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-pass-predictor-848ff9798	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-pass-predictor-848ff9798-tm765
20:16:51	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-848ff9798-tm765	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
20:16:52	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-pass-predictor-848ff9798-tm765	AddedInterface	Add eth0 [10.134.0.54/23] from ovn-kubernetes
20:16:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-848ff9798-tm765	Started	Started container storage-initializer
20:16:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-848ff9798-tm765	Created	Created container: storage-initializer
20:16:52	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-848ff9798-tm765	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
20:16:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-848ff9798-tm765	Started	Started container kube-rbac-proxy
20:16:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-848ff9798-tm765	Created	Created container: kube-rbac-proxy
20:16:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-848ff9798-tm765	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:16:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-848ff9798-tm765	Started	Started container kserve-container
20:16:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-848ff9798-tm765	Created	Created container: kserve-container
20:16:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-848ff9798-tm765	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1404" already present on machine
20:16:58 (x22)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:17:05	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:17:05	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:18:01	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-custom-pass] is Ready
20:18:06 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:18:06 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-custom-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:18:11 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-848ff9798-tm765	Unhealthy	Readiness probe failed: dial tcp 10.134.0.54:8080: connect: connection refused
20:18:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-848ff9798-tm765	Killing	Stopping container kserve-container
20:18:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-848ff9798-tm765	Killing	Stopping container kube-rbac-proxy
20:18:11	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-pass-predictor-848ff9798-tm765	Unhealthy	Readiness probe failed: Get "https://10.134.0.54:8643/healthz": dial tcp 10.134.0.54:8643: connect: connection refused
20:18:12	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-custom-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-custom-fail": the object has been modified; please apply your changes to the latest version and try again
20:18:12	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-custom-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-custom-fail-predictor-c7d959f56 from 0 to 1
20:18:12	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-custom-fail-predictor-c7d959f56	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-custom-fail-predictor-c7d959f56-zkxbc
20:18:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-c7d959f56-zkxbc	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
20:18:13	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-custom-fail-predictor-c7d959f56-zkxbc	AddedInterface	Add eth0 [10.134.0.55/23] from ovn-kubernetes
20:18:16 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-c7d959f56-zkxbc	Created	Created container: storage-initializer
20:18:16 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-c7d959f56-zkxbc	Started	Started container storage-initializer
20:18:16 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-c7d959f56-zkxbc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
20:18:17 (x18)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-custom-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:18:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-custom-fail-predictor-c7d959f56-zkxbc	Killing	Stopping container storage-initializer
20:18:23	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-pass": the object has been modified; please apply your changes to the latest version and try again
20:18:23 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-pass": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-pass": the object has been modified; please apply your changes to the latest version and try again
20:18:23	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-pass-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-pass-predictor-545f7c7957 from 0 to 1
20:18:23	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-pass-predictor-545f7c7957	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-pass-predictor-545f7c7957-66n4x
20:18:23	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-pass-predictor-545f7c7957-66n4x	AddedInterface	Add eth0 [10.134.0.56/23] from ovn-kubernetes
20:18:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-545f7c7957-66n4x	Started	Started container storage-initializer
20:18:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-545f7c7957-66n4x	Created	Created container: storage-initializer
20:18:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-545f7c7957-66n4x	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
20:18:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-545f7c7957-66n4x	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1404" already present on machine
20:18:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-545f7c7957-66n4x	Started	Started container kserve-container
20:18:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-545f7c7957-66n4x	Started	Started container kube-rbac-proxy
20:18:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-545f7c7957-66n4x	Created	Created container: kube-rbac-proxy
20:18:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-545f7c7957-66n4x	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:18:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-545f7c7957-66n4x	Created	Created container: kserve-container
20:18:29 (x22)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:19:08 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:19:08 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:19:22 (x8)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-545f7c7957-66n4x	Unhealthy	Readiness probe failed: dial tcp 10.134.0.56:8080: connect: connection refused
20:19:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:19:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-s3-tls-serving-pass-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:19:32	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-pass	InferenceServiceReady	InferenceService [isvc-sklearn-s3-tls-serving-pass] is Ready
20:19:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-545f7c7957-66n4x	Killing	Stopping container kserve-container
20:19:33	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-545f7c7957-66n4x	Killing	Stopping container kube-rbac-proxy
20:19:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-fail": the object has been modified; please apply your changes to the latest version and try again
20:19:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-7b6b7fcbd7-77h4d	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
20:19:34	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-s3-tls-serving-fail-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-s3-tls-serving-fail-predictor-7b6b7fcbd7 from 0 to 1
20:19:34 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-s3-tls-serving-fail": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-s3-tls-serving-fail": the object has been modified; please apply your changes to the latest version and try again
20:19:34	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-s3-tls-serving-fail-predictor-7b6b7fcbd7	SuccessfulCreate	Created pod: isvc-sklearn-s3-tls-serving-fail-predictor-7b6b7fcbd7-77h4d
20:19:35	kserve-ci-e2e-test	multus	isvc-sklearn-s3-tls-serving-fail-predictor-7b6b7fcbd7-77h4d	AddedInterface	Add eth0 [10.134.0.57/23] from ovn-kubernetes
20:19:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-pass-predictor-545f7c7957-66n4x	Unhealthy	Readiness probe failed: Get "https://10.134.0.56:8643/healthz": dial tcp 10.134.0.56:8643: connect: connection refused
20:19:38 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-7b6b7fcbd7-77h4d	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3460f014e7dc0a9d3daafe0716ca9eadf865f2892e0a5103d0b876da9f34891f" already present on machine
20:19:39 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-7b6b7fcbd7-77h4d	Created	Created container: storage-initializer
20:19:39 (x18)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-s3-tls-serving-fail	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:19:39 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-7b6b7fcbd7-77h4d	Started	Started container storage-initializer
20:19:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-s3-tls-serving-fail-predictor-7b6b7fcbd7-77h4d	Killing	Stopping container storage-initializer