Events

Time	Namespace	Component	RelatedObject	Reason	Message
21:07:38	kserve-ci-e2e-test		isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds to ip-10-0-135-188.ec2.internal
21:07:15	kserve-ci-e2e-test		isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf to ip-10-0-135-188.ec2.internal
21:07:54	kserve-ci-e2e-test		raw-sklearn-92cf8-predictor-d4757fc6b-s49vs	Scheduled	Successfully assigned kserve-ci-e2e-test/raw-sklearn-92cf8-predictor-d4757fc6b-s49vs to ip-10-0-135-188.ec2.internal
21:01:14	kserve-ci-e2e-test		model-chainer-raw-hpa-1851d-c96567755-bzhlm	Scheduled	Successfully assigned kserve-ci-e2e-test/model-chainer-raw-hpa-1851d-c96567755-bzhlm to ip-10-0-137-220.ec2.internal
21:05:55	kserve-ci-e2e-test		isvc-primary-77d927-predictor-86f44cb6d7-qvq7f	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-primary-77d927-predictor-86f44cb6d7-qvq7f to ip-10-0-135-188.ec2.internal
20:55:59	kserve-ci-e2e-test		isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh to ip-10-0-135-188.ec2.internal
21:09:19	kserve-ci-e2e-test		raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc	Scheduled	Successfully assigned kserve-ci-e2e-test/raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc to ip-10-0-135-188.ec2.internal
20:59:34	kserve-ci-e2e-test		isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r to ip-10-0-135-188.ec2.internal
20:59:24	kserve-ci-e2e-test		model-chainer-raw-c682d-544fdc8c65-64hj5	Scheduled	Successfully assigned kserve-ci-e2e-test/model-chainer-raw-c682d-544fdc8c65-64hj5 to ip-10-0-137-220.ec2.internal
21:03:20	kserve-ci-e2e-test		isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2 to ip-10-0-135-188.ec2.internal
20:57:44	kserve-ci-e2e-test		isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c to ip-10-0-135-188.ec2.internal
21:01:45	kserve-ci-e2e-test		isvc-logger-raw-1255c-predictor-65576c9ff7-gl997	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-1255c-predictor-65576c9ff7-gl997 to ip-10-0-135-188.ec2.internal
20:59:34	kserve-ci-e2e-test		isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn to ip-10-0-135-188.ec2.internal
20:57:44	kserve-ci-e2e-test		isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm to ip-10-0-135-188.ec2.internal
21:01:25	kserve-ci-e2e-test		message-dumper-raw-1255c-predictor-cc944bc4-sxm5p	Scheduled	Successfully assigned kserve-ci-e2e-test/message-dumper-raw-1255c-predictor-cc944bc4-sxm5p to ip-10-0-135-188.ec2.internal
20:55:59 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-raw-sklearn-batcher-06549	UpdateFailed	Failed to update status for InferenceService "isvc-raw-sklearn-batcher-06549": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-06549": the object has been modified; please apply your changes to the latest version and try again
20:55:59	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh	Started	Started container storage-initializer
20:55:59	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh	Created	Created container: storage-initializer
20:55:59 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-raw-sklearn-batcher-06549	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-06549": the object has been modified; please apply your changes to the latest version and try again
20:55:59	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine
20:55:59	kserve-ci-e2e-test	deployment-controller	isvc-raw-sklearn-batcher-06549-predictor	ScalingReplicaSet	Scaled up replica set isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8 from 0 to 1
20:55:59	kserve-ci-e2e-test	replicaset-controller	isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8	SuccessfulCreate	Created pod: isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh
20:55:59	kserve-ci-e2e-test	multus	isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh	AddedInterface	Add eth0 [10.134.0.19/23] from ovn-kubernetes
20:56:03	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1683"
20:56:18	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh	Started	Started container kserve-container
20:56:18	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1683" in 14.431s (14.431s including waiting). Image size: 1555648931 bytes.
20:56:18	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh	Created	Created container: kserve-container
20:56:18	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
20:56:20	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.343s (2.343s including waiting). Image size: 211946088 bytes.
20:56:20	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh	Created	Created container: kube-rbac-proxy
20:56:20	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh	Started	Started container kube-rbac-proxy
20:56:20	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh	Pulling	Pulling image "quay.io/opendatahub/kserve-agent@sha256:eb91db6613fb4bde95991e8e02d444548e2e6e23bd5e4e0a11c1f350f3203183"
20:56:22	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh	Created	Created container: agent
20:56:22	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh	Started	Started container agent
20:56:22	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:eb91db6613fb4bde95991e8e02d444548e2e6e23bd5e4e0a11c1f350f3203183" in 2.245s (2.245s including waiting). Image size: 239994455 bytes.
20:56:59 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-raw-sklearn-batcher-06549-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:56:59 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-raw-sklearn-batcher-06549-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:57:29	kserve-ci-e2e-test	v1beta1Controllers	isvc-raw-sklearn-batcher-06549	InferenceServiceReady	InferenceService [isvc-raw-sklearn-batcher-06549] is Ready
20:57:29 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-raw-sklearn-batcher-06549	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:57:44 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-raw-sklearn-batcher-06549-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:57:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine
20:57:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-graph-raw-c682d	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-graph-raw-c682d": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-graph-raw-c682d": the object has been modified; please apply your changes to the latest version and try again
20:57:44	kserve-ci-e2e-test	multus	isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c	AddedInterface	Add eth0 [10.134.0.20/23] from ovn-kubernetes
20:57:44 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-raw-sklearn-batcher-06549-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:57:44	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh	Killing	Stopping container kube-rbac-proxy
20:57:44	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554	SuccessfulCreate	Created pod: isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c
20:57:44	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-graph-raw-c682d-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-graph-raw-c682d-predictor-6c549b746f from 0 to 1
20:57:44	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-graph-raw-c682d-predictor-6c549b746f	SuccessfulCreate	Created pod: isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm
20:57:44	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh	Killing	Stopping container agent
20:57:44	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-graph-raw-c682d-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554 from 0 to 1
20:57:44	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-raw-c682d	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-graph-raw-c682d": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-c682d": the object has been modified; please apply your changes to the latest version and try again
20:57:44	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh	Killing	Stopping container kserve-container
20:57:45	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm	Created	Created container: storage-initializer
20:57:45	kserve-ci-e2e-test	multus	isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm	AddedInterface	Add eth0 [10.134.0.21/23] from ovn-kubernetes
20:57:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c	Started	Started container storage-initializer
20:57:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c	Created	Created container: storage-initializer
20:57:45	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine
20:57:45	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm	Started	Started container storage-initializer
20:57:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm	Pulling	Pulling image "kserve/xgbserver:latest"
20:57:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c	Started	Started container kube-rbac-proxy
20:57:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c	Created	Created container: kserve-container
20:57:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c	Created	Created container: kube-rbac-proxy
20:57:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1683" already present on machine
20:57:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:57:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c	Started	Started container kserve-container
20:57:59 (x10)	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh	Unhealthy	Readiness probe failed: dial tcp 10.134.0.19:8080: connect: connection refused
20:57:59 (x10)	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
20:58:04 (x5)	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-06549-predictor-5778c86cc8-6wvjh	Unhealthy	Readiness probe failed: Get "https://10.134.0.19:8643/healthz": dial tcp 10.134.0.19:8643: connect: connection refused
20:58:07	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm	Started	Started container kube-rbac-proxy
20:58:07	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 18.121s (18.121s including waiting). Image size: 1305293052 bytes.
20:58:07	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm	Created	Created container: kserve-container
20:58:07	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm	Started	Started container kserve-container
20:58:07	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:58:07	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm	Created	Created container: kube-rbac-proxy
20:58:29 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-raw-c682d-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:58:29 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-raw-c682d-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:58:44 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-raw-c682d-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:58:44 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-raw-c682d-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:59:05 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-graph-raw-c682d	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:59:05	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-graph-raw-c682d	InferenceServiceReady	InferenceService [isvc-xgboost-graph-raw-c682d] is Ready
20:59:07 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-raw-c682d	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:59:07	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-raw-c682d	InferenceServiceReady	InferenceService [isvc-sklearn-graph-raw-c682d] is Ready
20:59:24	kserve-ci-e2e-test	replicaset-controller	model-chainer-raw-c682d-544fdc8c65	SuccessfulCreate	Created pod: model-chainer-raw-c682d-544fdc8c65-64hj5
20:59:24	kserve-ci-e2e-test	deployment-controller	model-chainer-raw-c682d	ScalingReplicaSet	Scaled up replica set model-chainer-raw-c682d-544fdc8c65 from 0 to 1
20:59:24	kserve-ci-e2e-test	kubelet	model-chainer-raw-c682d-544fdc8c65-64hj5	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-raw-c682d-serving-cert" not found
20:59:24	kserve-ci-e2e-test	InferenceGraphController	model-chainer-raw-c682d	InferenceGraphReady	InferenceGraph [model-chainer-raw-c682d] is Ready
20:59:25	kserve-ci-e2e-test	kubelet	model-chainer-raw-c682d-544fdc8c65-64hj5	Pulling	Pulling image "quay.io/opendatahub/kserve-router@sha256:78561fce36825084e3c48d0137c513c345701cffde73e206abb14ec219f8181b"
20:59:25	kserve-ci-e2e-test	multus	model-chainer-raw-c682d-544fdc8c65-64hj5	AddedInterface	Add eth0 [10.133.0.30/23] from ovn-kubernetes
20:59:28	kserve-ci-e2e-test	kubelet	model-chainer-raw-c682d-544fdc8c65-64hj5	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:78561fce36825084e3c48d0137c513c345701cffde73e206abb14ec219f8181b" in 2.699s (2.699s including waiting). Image size: 218070105 bytes.
20:59:28	kserve-ci-e2e-test	kubelet	model-chainer-raw-c682d-544fdc8c65-64hj5	Created	Created container: model-chainer-raw-c682d
20:59:28	kserve-ci-e2e-test	kubelet	model-chainer-raw-c682d-544fdc8c65-64hj5	Started	Started container model-chainer-raw-c682d
20:59:29 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-raw-c682d-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:59:29 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-raw-c682d-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:59:29 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-raw-c682d-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:59:29 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-raw-c682d-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:59:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c	Killing	Stopping container kserve-container
20:59:34	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c	Killing	Stopping container kube-rbac-proxy
20:59:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-graph-raw-hpa-1851d	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-graph-raw-hpa-1851d": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-graph-raw-hpa-1851d": the object has been modified; please apply your changes to the latest version and try again
20:59:34	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957	SuccessfulCreate	Created pod: isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn
20:59:34	kserve-ci-e2e-test	kubelet	model-chainer-raw-c682d-544fdc8c65-64hj5	Killing	Stopping container model-chainer-raw-c682d
20:59:34	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-raw-hpa-1851d	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-graph-raw-hpa-1851d": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-hpa-1851d": the object has been modified; please apply your changes to the latest version and try again
20:59:34	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-graph-raw-hpa-1851d-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957 from 0 to 1
20:59:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm	Killing	Stopping container kube-rbac-proxy
20:59:34	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-graph-raw-hpa-1851d-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4 from 0 to 1
20:59:34	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4	SuccessfulCreate	Created pod: isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r
20:59:34	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm	Killing	Stopping container kserve-container
20:59:35	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn	Created	Created container: storage-initializer
20:59:35	kserve-ci-e2e-test	multus	isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn	AddedInterface	Add eth0 [10.134.0.23/23] from ovn-kubernetes
20:59:35	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn	Started	Started container storage-initializer
20:59:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r	Started	Started container storage-initializer
20:59:35	kserve-ci-e2e-test	multus	isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r	AddedInterface	Add eth0 [10.134.0.22/23] from ovn-kubernetes
20:59:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r	Created	Created container: storage-initializer
20:59:35	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine
20:59:35	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine
20:59:35 (x8)	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm	Unhealthy	Readiness probe failed: dial tcp 10.134.0.21:8080: connect: connection refused
20:59:35	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-c682d-predictor-6c549b746f-frbjm	Unhealthy	Readiness probe failed: Get "https://10.134.0.21:8643/healthz": dial tcp 10.134.0.21:8643: connect: connection refused
20:59:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c	Unhealthy	Readiness probe failed: Get "https://10.134.0.20:8643/healthz": dial tcp 10.134.0.20:8643: connect: connection refused
20:59:37 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-c682d-predictor-6c6fb84554-mj72c	Unhealthy	Readiness probe failed: dial tcp 10.134.0.20:8080: connect: connection refused
20:59:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r	Created	Created container: kube-rbac-proxy
20:59:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:59:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r	Started	Started container kube-rbac-proxy
20:59:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1683" already present on machine
20:59:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn	Pulled	Container image "kserve/xgbserver:latest" already present on machine
20:59:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r	Created	Created container: kserve-container
20:59:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn	Created	Created container: kserve-container
20:59:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn	Started	Started container kserve-container
20:59:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r	Started	Started container kserve-container
20:59:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:59:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn	Created	Created container: kube-rbac-proxy
20:59:39	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn	Started	Started container kube-rbac-proxy
21:00:04 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-raw-hpa-1851d-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:00:04 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-raw-hpa-1851d-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:00:04 (x7)	kserve-ci-e2e-test	kubelet	model-chainer-raw-c682d-544fdc8c65-64hj5	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
21:00:04 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-raw-hpa-1851d-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:00:04 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-raw-hpa-1851d-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:00:36 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-graph-raw-hpa-1851d	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:00:36	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-graph-raw-hpa-1851d	InferenceServiceReady	InferenceService [isvc-xgboost-graph-raw-hpa-1851d] is Ready
21:00:49 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-raw-hpa-1851d-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:00:49 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-raw-hpa-1851d-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:00:57	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-raw-hpa-1851d	InferenceServiceReady	InferenceService [isvc-sklearn-graph-raw-hpa-1851d] is Ready
21:00:57 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-raw-hpa-1851d	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:01:14	kserve-ci-e2e-test	InferenceGraphController	model-chainer-raw-hpa-1851d	InferenceGraphReady	InferenceGraph [model-chainer-raw-hpa-1851d] is Ready
21:01:14	kserve-ci-e2e-test	deployment-controller	model-chainer-raw-hpa-1851d	ScalingReplicaSet	Scaled up replica set model-chainer-raw-hpa-1851d-c96567755 from 0 to 1
21:01:14	kserve-ci-e2e-test	replicaset-controller	model-chainer-raw-hpa-1851d-c96567755	SuccessfulCreate	Created pod: model-chainer-raw-hpa-1851d-c96567755-bzhlm
21:01:15	kserve-ci-e2e-test	kubelet	model-chainer-raw-hpa-1851d-c96567755-bzhlm	Created	Created container: model-chainer-raw-hpa-1851d
21:01:15	kserve-ci-e2e-test	kubelet	model-chainer-raw-hpa-1851d-c96567755-bzhlm	Started	Started container model-chainer-raw-hpa-1851d
21:01:15	kserve-ci-e2e-test	multus	model-chainer-raw-hpa-1851d-c96567755-bzhlm	AddedInterface	Add eth0 [10.133.0.31/23] from ovn-kubernetes
21:01:15	kserve-ci-e2e-test	kubelet	model-chainer-raw-hpa-1851d-c96567755-bzhlm	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:78561fce36825084e3c48d0137c513c345701cffde73e206abb14ec219f8181b" already present on machine
21:01:19 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-raw-hpa-1851d-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:01:19 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-raw-hpa-1851d-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:01:24	kserve-ci-e2e-test	v1beta1Controllers	message-dumper-raw-1255c	UpdateFailed	Failed to update status for InferenceService "message-dumper-raw-1255c": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-1255c": the object has been modified; please apply your changes to the latest version and try again
21:01:24	kserve-ci-e2e-test	kubelet	model-chainer-raw-hpa-1851d-c96567755-bzhlm	Killing	Stopping container model-chainer-raw-hpa-1851d
21:01:24	kserve-ci-e2e-test	deployment-controller	message-dumper-raw-1255c-predictor	ScalingReplicaSet	Scaled up replica set message-dumper-raw-1255c-predictor-cc944bc4 from 0 to 1
21:01:24	kserve-ci-e2e-test	replicaset-controller	message-dumper-raw-1255c-predictor-cc944bc4	SuccessfulCreate	Created pod: message-dumper-raw-1255c-predictor-cc944bc4-sxm5p
21:01:25	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn	Killing	Stopping container kube-rbac-proxy
21:01:25	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r	Killing	Stopping container kube-rbac-proxy
21:01:25	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r	Killing	Stopping container kserve-container
21:01:25	kserve-ci-e2e-test	kubelet	message-dumper-raw-1255c-predictor-cc944bc4-sxm5p	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "message-dumper-raw-1255c-predictor-serving-cert" not found
21:01:25	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn	Killing	Stopping container kserve-container
21:01:26	kserve-ci-e2e-test	multus	message-dumper-raw-1255c-predictor-cc944bc4-sxm5p	AddedInterface	Add eth0 [10.134.0.24/23] from ovn-kubernetes
21:01:26	kserve-ci-e2e-test	kubelet	message-dumper-raw-1255c-predictor-cc944bc4-sxm5p	Pulling	Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display"
21:01:26 (x8)	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn	Unhealthy	Readiness probe failed: dial tcp 10.134.0.23:8080: connect: connection refused
21:01:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-1851d-predictor-6cf48c7957-4tcvn	Unhealthy	Readiness probe failed: Get "https://10.134.0.23:8643/healthz": dial tcp 10.134.0.23:8643: connect: connection refused
21:01:27	kserve-ci-e2e-test	kubelet	message-dumper-raw-1255c-predictor-cc944bc4-sxm5p	Created	Created container: kube-rbac-proxy
21:01:27	kserve-ci-e2e-test	kubelet	message-dumper-raw-1255c-predictor-cc944bc4-sxm5p	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:01:27	kserve-ci-e2e-test	kubelet	message-dumper-raw-1255c-predictor-cc944bc4-sxm5p	Started	Started container kserve-container
21:01:27	kserve-ci-e2e-test	kubelet	message-dumper-raw-1255c-predictor-cc944bc4-sxm5p	Created	Created container: kserve-container
21:01:27	kserve-ci-e2e-test	kubelet	message-dumper-raw-1255c-predictor-cc944bc4-sxm5p	Pulled	Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.175s (1.175s including waiting). Image size: 14813193 bytes.
21:01:27	kserve-ci-e2e-test	kubelet	message-dumper-raw-1255c-predictor-cc944bc4-sxm5p	Started	Started container kube-rbac-proxy
21:01:27 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r	Unhealthy	Readiness probe failed: dial tcp 10.134.0.22:8080: connect: connection refused
21:01:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-1851d-predictor-797cddf7c4-bvs8r	Unhealthy	Readiness probe failed: Get "https://10.134.0.22:8643/healthz": dial tcp 10.134.0.22:8643: connect: connection refused
21:01:35	kserve-ci-e2e-test	v1beta1Controllers	message-dumper-raw-1255c	InferenceServiceReady	InferenceService [message-dumper-raw-1255c] is Ready
21:01:35 (x7)	kserve-ci-e2e-test	v1beta1Controllers	message-dumper-raw-1255c	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:01:44	kserve-ci-e2e-test	deployment-controller	isvc-logger-raw-1255c-predictor	ScalingReplicaSet	Scaled up replica set isvc-logger-raw-1255c-predictor-65576c9ff7 from 0 to 1
21:01:45	kserve-ci-e2e-test	multus	isvc-logger-raw-1255c-predictor-65576c9ff7-gl997	AddedInterface	Add eth0 [10.134.0.25/23] from ovn-kubernetes
21:01:45	kserve-ci-e2e-test	kubelet	isvc-logger-raw-1255c-predictor-65576c9ff7-gl997	Started	Started container storage-initializer
21:01:45	kserve-ci-e2e-test	replicaset-controller	isvc-logger-raw-1255c-predictor-65576c9ff7	SuccessfulCreate	Created pod: isvc-logger-raw-1255c-predictor-65576c9ff7-gl997
21:01:45 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger-raw-1255c	UpdateFailed	Failed to update status for InferenceService "isvc-logger-raw-1255c": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-1255c": the object has been modified; please apply your changes to the latest version and try again
21:01:45	kserve-ci-e2e-test	kubelet	isvc-logger-raw-1255c-predictor-65576c9ff7-gl997	Created	Created container: storage-initializer
21:01:45	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger-raw-1255c	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-1255c": the object has been modified; please apply your changes to the latest version and try again
21:01:45	kserve-ci-e2e-test	kubelet	isvc-logger-raw-1255c-predictor-65576c9ff7-gl997	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine
21:01:49	kserve-ci-e2e-test	kubelet	isvc-logger-raw-1255c-predictor-65576c9ff7-gl997	Started	Started container agent
21:01:49	kserve-ci-e2e-test	kubelet	isvc-logger-raw-1255c-predictor-65576c9ff7-gl997	Created	Created container: kserve-container
21:01:49	kserve-ci-e2e-test	kubelet	isvc-logger-raw-1255c-predictor-65576c9ff7-gl997	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:eb91db6613fb4bde95991e8e02d444548e2e6e23bd5e4e0a11c1f350f3203183" already present on machine
21:01:49	kserve-ci-e2e-test	kubelet	isvc-logger-raw-1255c-predictor-65576c9ff7-gl997	Started	Started container kube-rbac-proxy
21:01:49	kserve-ci-e2e-test	kubelet	isvc-logger-raw-1255c-predictor-65576c9ff7-gl997	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1683" already present on machine
21:01:49	kserve-ci-e2e-test	kubelet	isvc-logger-raw-1255c-predictor-65576c9ff7-gl997	Created	Created container: kube-rbac-proxy
21:01:49	kserve-ci-e2e-test	kubelet	isvc-logger-raw-1255c-predictor-65576c9ff7-gl997	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:01:49	kserve-ci-e2e-test	kubelet	isvc-logger-raw-1255c-predictor-65576c9ff7-gl997	Created	Created container: agent
21:01:49	kserve-ci-e2e-test	kubelet	isvc-logger-raw-1255c-predictor-65576c9ff7-gl997	Started	Started container kserve-container
21:01:52 (x6)	kserve-ci-e2e-test	kubelet	model-chainer-raw-hpa-1851d-c96567755-bzhlm	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
21:01:54 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-raw-1255c-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:01:54 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-raw-1255c-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:02:29 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-raw-1255c-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:02:29 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-raw-1255c-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:03:06 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger-raw-1255c	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:03:06	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger-raw-1255c	InferenceServiceReady	InferenceService [isvc-logger-raw-1255c] is Ready
21:03:15 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-raw-1255c-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:03:15 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-raw-1255c-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:03:20	kserve-ci-e2e-test	kubelet	isvc-logger-raw-1255c-predictor-65576c9ff7-gl997	Killing	Stopping container agent
21:03:20	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644	SuccessfulCreate	Created pod: isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2
21:03:20	kserve-ci-e2e-test	multus	isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2	AddedInterface	Add eth0 [10.134.0.26/23] from ovn-kubernetes
21:03:20	kserve-ci-e2e-test	kubelet	message-dumper-raw-1255c-predictor-cc944bc4-sxm5p	Killing	Stopping container kube-rbac-proxy
21:03:20	kserve-ci-e2e-test	kubelet	message-dumper-raw-1255c-predictor-cc944bc4-sxm5p	Killing	Stopping container kserve-container
21:03:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine
21:03:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2	Created	Created container: storage-initializer
21:03:20	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2	Started	Started container storage-initializer
21:03:20	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-scale-raw-a940b-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644 from 0 to 1
21:03:20	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scale-raw-a940b	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-scale-raw-a940b": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scale-raw-a940b": the object has been modified; please apply your changes to the latest version and try again
21:03:20	kserve-ci-e2e-test	kubelet	isvc-logger-raw-1255c-predictor-65576c9ff7-gl997	Killing	Stopping container kserve-container
21:03:20	kserve-ci-e2e-test	kubelet	isvc-logger-raw-1255c-predictor-65576c9ff7-gl997	Killing	Stopping container kube-rbac-proxy
21:03:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2	Created	Created container: kube-rbac-proxy
21:03:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2	Started	Started container kube-rbac-proxy
21:03:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:03:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2	Started	Started container kserve-container
21:03:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2	Created	Created container: kserve-container
21:03:24	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1683" already present on machine
21:03:26 (x10)	kserve-ci-e2e-test	kubelet	isvc-logger-raw-1255c-predictor-65576c9ff7-gl997	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
21:03:32 (x3)	kserve-ci-e2e-test	kubelet	isvc-logger-raw-1255c-predictor-65576c9ff7-gl997	Unhealthy	Readiness probe failed: Get "https://10.134.0.25:8643/healthz": dial tcp 10.134.0.25:8643: connect: connection refused
21:03:35	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-scale-raw-a940b-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:03:35	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-scale-raw-a940b-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:03:36 (x12)	kserve-ci-e2e-test	kubelet	isvc-logger-raw-1255c-predictor-65576c9ff7-gl997	Unhealthy	Readiness probe failed: dial tcp 10.134.0.25:8080: connect: connection refused
21:05:49 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scale-raw-a940b	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:05:49	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scale-raw-a940b	InferenceServiceReady	InferenceService [isvc-sklearn-scale-raw-a940b] is Ready
21:05:50 (x9)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-scale-raw-a940b-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:05:50 (x9)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-scale-raw-a940b-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:05:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2	Killing	Stopping container kube-rbac-proxy
21:05:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2	Killing	Stopping container kserve-container
21:05:55	kserve-ci-e2e-test	replicaset-controller	isvc-primary-77d927-predictor-86f44cb6d7	SuccessfulCreate	Created pod: isvc-primary-77d927-predictor-86f44cb6d7-qvq7f
21:05:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-77d927	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-77d927": the object has been modified; please apply your changes to the latest version and try again
21:05:55	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-77d927	UpdateFailed	Failed to update status for InferenceService "isvc-primary-77d927": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-77d927": the object has been modified; please apply your changes to the latest version and try again
21:05:55	kserve-ci-e2e-test	deployment-controller	isvc-primary-77d927-predictor	ScalingReplicaSet	Scaled up replica set isvc-primary-77d927-predictor-86f44cb6d7 from 0 to 1
21:05:56	kserve-ci-e2e-test	multus	isvc-primary-77d927-predictor-86f44cb6d7-qvq7f	AddedInterface	Add eth0 [10.134.0.27/23] from ovn-kubernetes
21:05:56	kserve-ci-e2e-test	kubelet	isvc-primary-77d927-predictor-86f44cb6d7-qvq7f	Started	Started container storage-initializer
21:05:56	kserve-ci-e2e-test	kubelet	isvc-primary-77d927-predictor-86f44cb6d7-qvq7f	Created	Created container: storage-initializer
21:05:56	kserve-ci-e2e-test	kubelet	isvc-primary-77d927-predictor-86f44cb6d7-qvq7f	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine
21:05:57 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2	Unhealthy	Readiness probe failed: Get "https://10.134.0.26:8643/healthz": dial tcp 10.134.0.26:8643: connect: connection refused
21:05:59 (x18)	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-a940b-predictor-7fbb4f6644-twvm2	Unhealthy	Readiness probe failed: dial tcp 10.134.0.26:8080: connect: connection refused
21:06:00	kserve-ci-e2e-test	kubelet	isvc-primary-77d927-predictor-86f44cb6d7-qvq7f	Created	Created container: kube-rbac-proxy
21:06:00	kserve-ci-e2e-test	kubelet	isvc-primary-77d927-predictor-86f44cb6d7-qvq7f	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:06:00	kserve-ci-e2e-test	kubelet	isvc-primary-77d927-predictor-86f44cb6d7-qvq7f	Started	Started container kserve-container
21:06:00	kserve-ci-e2e-test	kubelet	isvc-primary-77d927-predictor-86f44cb6d7-qvq7f	Started	Started container kube-rbac-proxy
21:06:00	kserve-ci-e2e-test	kubelet	isvc-primary-77d927-predictor-86f44cb6d7-qvq7f	Created	Created container: kserve-container
21:06:00	kserve-ci-e2e-test	kubelet	isvc-primary-77d927-predictor-86f44cb6d7-qvq7f	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1683" already present on machine
21:06:10	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-77d927-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:06:10	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-77d927-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:07:07 (x9)	kserve-ci-e2e-test	kubelet	isvc-primary-77d927-predictor-86f44cb6d7-qvq7f	Unhealthy	Readiness probe failed: dial tcp 10.134.0.27:8080: connect: connection refused
21:07:11 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-77d927	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:07:11	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-77d927	InferenceServiceReady	InferenceService [isvc-primary-77d927] is Ready
21:07:15	kserve-ci-e2e-test	replicaset-controller	isvc-secondary-77d927-predictor-67bbb9f4c6	SuccessfulCreate	Created pod: isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf
21:07:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-77d927	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-77d927": the object has been modified; please apply your changes to the latest version and try again
21:07:15	kserve-ci-e2e-test	deployment-controller	isvc-secondary-77d927-predictor	ScalingReplicaSet	Scaled up replica set isvc-secondary-77d927-predictor-67bbb9f4c6 from 0 to 1
21:07:15	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-77d927	UpdateFailed	Failed to update status for InferenceService "isvc-secondary-77d927": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-77d927": the object has been modified; please apply your changes to the latest version and try again
21:07:16	kserve-ci-e2e-test	multus	isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf	AddedInterface	Add eth0 [10.134.0.28/23] from ovn-kubernetes
21:07:21 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine
21:07:21 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf	Created	Created container: storage-initializer
21:07:21 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf	Started	Started container storage-initializer
21:07:25 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-77d927-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:07:25 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-77d927-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:07:27	kserve-ci-e2e-test	kubelet	isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf	BackOff	Back-off restarting failed container storage-initializer in pod isvc-secondary-77d927-predictor-67bbb9f4c6-cf6xf_kserve-ci-e2e-test(f2881f6d-1c8b-4dcf-89b8-bc38d9de1f02)
21:07:27 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-77d927	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:07:30	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-77d927-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:07:30	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-77d927-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:07:33	kserve-ci-e2e-test	kubelet	isvc-primary-77d927-predictor-86f44cb6d7-qvq7f	Killing	Stopping container kube-rbac-proxy
21:07:33	kserve-ci-e2e-test	kubelet	isvc-primary-77d927-predictor-86f44cb6d7-qvq7f	Killing	Stopping container kserve-container
21:07:37	kserve-ci-e2e-test	kubelet	isvc-primary-77d927-predictor-86f44cb6d7-qvq7f	Unhealthy	Readiness probe failed: Get "https://10.134.0.27:8643/healthz": dial tcp 10.134.0.27:8643: connect: connection refused
21:07:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-2dd56b	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-2dd56b": the object has been modified; please apply your changes to the latest version and try again
21:07:38	kserve-ci-e2e-test	replicaset-controller	isvc-init-fail-2dd56b-predictor-7fd4584776	SuccessfulCreate	Created pod: isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds
21:07:38	kserve-ci-e2e-test	deployment-controller	isvc-init-fail-2dd56b-predictor	ScalingReplicaSet	Scaled up replica set isvc-init-fail-2dd56b-predictor-7fd4584776 from 0 to 1
21:07:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-2dd56b	UpdateFailed	Failed to update status for InferenceService "isvc-init-fail-2dd56b": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-2dd56b": the object has been modified; please apply your changes to the latest version and try again
21:07:39	kserve-ci-e2e-test	multus	isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds	AddedInterface	Add eth0 [10.134.0.29/23] from ovn-kubernetes
21:07:44 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds	Started	Started container storage-initializer
21:07:44 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-2dd56b	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:07:44 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine
21:07:44 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds	Created	Created container: storage-initializer
21:07:48	kserve-ci-e2e-test	kubelet	isvc-init-fail-2dd56b-predictor-7fd4584776-k9xds	Killing	Stopping container storage-initializer
21:07:54	kserve-ci-e2e-test	kubelet	raw-sklearn-92cf8-predictor-d4757fc6b-s49vs	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "raw-sklearn-92cf8-predictor-serving-cert" not found
21:07:54	kserve-ci-e2e-test	v1beta1Controllers	raw-sklearn-92cf8	UpdateFailed	Failed to update status for InferenceService "raw-sklearn-92cf8": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-92cf8": the object has been modified; please apply your changes to the latest version and try again
21:07:54	kserve-ci-e2e-test	deployment-controller	raw-sklearn-92cf8-predictor	ScalingReplicaSet	Scaled up replica set raw-sklearn-92cf8-predictor-d4757fc6b from 0 to 1
21:07:54	kserve-ci-e2e-test	replicaset-controller	raw-sklearn-92cf8-predictor-d4757fc6b	SuccessfulCreate	Created pod: raw-sklearn-92cf8-predictor-d4757fc6b-s49vs
21:07:55	kserve-ci-e2e-test	kubelet	raw-sklearn-92cf8-predictor-d4757fc6b-s49vs	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine
21:07:55	kserve-ci-e2e-test	kubelet	raw-sklearn-92cf8-predictor-d4757fc6b-s49vs	Created	Created container: storage-initializer
21:07:55	kserve-ci-e2e-test	kubelet	raw-sklearn-92cf8-predictor-d4757fc6b-s49vs	Started	Started container storage-initializer
21:07:55	kserve-ci-e2e-test	multus	raw-sklearn-92cf8-predictor-d4757fc6b-s49vs	AddedInterface	Add eth0 [10.134.0.30/23] from ovn-kubernetes
21:07:59	kserve-ci-e2e-test	kubelet	raw-sklearn-92cf8-predictor-d4757fc6b-s49vs	Created	Created container: kube-rbac-proxy
21:07:59	kserve-ci-e2e-test	kubelet	raw-sklearn-92cf8-predictor-d4757fc6b-s49vs	Started	Started container kube-rbac-proxy
21:07:59	kserve-ci-e2e-test	kubelet	raw-sklearn-92cf8-predictor-d4757fc6b-s49vs	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1683" already present on machine
21:07:59	kserve-ci-e2e-test	kubelet	raw-sklearn-92cf8-predictor-d4757fc6b-s49vs	Created	Created container: kserve-container
21:07:59	kserve-ci-e2e-test	kubelet	raw-sklearn-92cf8-predictor-d4757fc6b-s49vs	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:07:59	kserve-ci-e2e-test	kubelet	raw-sklearn-92cf8-predictor-d4757fc6b-s49vs	Started	Started container kserve-container
21:08:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	raw-sklearn-92cf8-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:08:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	raw-sklearn-92cf8-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:09:07	kserve-ci-e2e-test	v1beta1Controllers	raw-sklearn-92cf8	InferenceServiceReady	InferenceService [raw-sklearn-92cf8] is Ready
21:09:07 (x11)	kserve-ci-e2e-test	v1beta1Controllers	raw-sklearn-92cf8	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:09:09 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	raw-sklearn-92cf8-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:09:09 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	raw-sklearn-92cf8-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:09:14	kserve-ci-e2e-test	kubelet	raw-sklearn-92cf8-predictor-d4757fc6b-s49vs	Killing	Stopping container kube-rbac-proxy
21:09:14	kserve-ci-e2e-test	kubelet	raw-sklearn-92cf8-predictor-d4757fc6b-s49vs	Killing	Stopping container kserve-container
21:09:17	kserve-ci-e2e-test	kubelet	raw-sklearn-92cf8-predictor-d4757fc6b-s49vs	Unhealthy	Readiness probe failed: Get "https://10.134.0.30:8643/healthz": dial tcp 10.134.0.30:8643: connect: connection refused
21:09:17 (x9)	kserve-ci-e2e-test	kubelet	raw-sklearn-92cf8-predictor-d4757fc6b-s49vs	Unhealthy	Readiness probe failed: dial tcp 10.134.0.30:8080: connect: connection refused
21:09:19	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc	Started	Started container storage-initializer
21:09:19	kserve-ci-e2e-test	replicaset-controller	raw-sklearn-runtime-830cb-predictor-56b49bc498	SuccessfulCreate	Created pod: raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc
21:09:19	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc	Created	Created container: storage-initializer
21:09:19	kserve-ci-e2e-test	multus	raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc	AddedInterface	Add eth0 [10.134.0.31/23] from ovn-kubernetes
21:09:19	kserve-ci-e2e-test	v1beta1Controllers	raw-sklearn-runtime-830cb	UpdateFailed	Failed to update status for InferenceService "raw-sklearn-runtime-830cb": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-830cb": the object has been modified; please apply your changes to the latest version and try again
21:09:19	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine
21:09:19	kserve-ci-e2e-test	deployment-controller	raw-sklearn-runtime-830cb-predictor	ScalingReplicaSet	Scaled up replica set raw-sklearn-runtime-830cb-predictor-56b49bc498 from 0 to 1
21:09:24	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1683" already present on machine
21:09:24	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc	Created	Created container: kserve-container
21:09:24	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc	Started	Started container kserve-container
21:09:24	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:09:24	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc	Created	Created container: kube-rbac-proxy
21:09:24	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc	Started	Started container kube-rbac-proxy
21:09:49 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	raw-sklearn-runtime-830cb-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:09:49 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	raw-sklearn-runtime-830cb-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:10:41 (x11)	kserve-ci-e2e-test	v1beta1Controllers	raw-sklearn-runtime-830cb	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:10:41	kserve-ci-e2e-test	v1beta1Controllers	raw-sklearn-runtime-830cb	InferenceServiceReady	InferenceService [raw-sklearn-runtime-830cb] is Ready
21:10:49	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc	Killing	Stopping container kube-rbac-proxy
21:10:49	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc	Killing	Stopping container kserve-container
21:10:49 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	raw-sklearn-runtime-830cb-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:10:49 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	raw-sklearn-runtime-830cb-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:10:51	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc	Unhealthy	Readiness probe failed: Get "https://10.134.0.31:8643/healthz": dial tcp 10.134.0.31:8643: connect: connection refused
21:10:51 (x10)	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-830cb-predictor-56b49bc498-h2rwc	Unhealthy	Readiness probe failed: dial tcp 10.134.0.31:8080: connect: connection refused