Events

Time	Namespace	Component	RelatedObject	Reason	Message
20:55:17	kserve-ci-e2e-test		isvc-sklearn-batcher-predictor-7595457646-gz6j8	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7595457646-gz6j8 to ip-10-0-129-137.ec2.internal
20:57:02	kserve-ci-e2e-test		isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj to ip-10-0-129-137.ec2.internal
21:09:24	kserve-ci-e2e-test		isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh to ip-10-0-129-137.ec2.internal
21:05:53	kserve-ci-e2e-test		isvc-autogluon-v2-variants-predictor-c6767c94c-6d854	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-6d854 to ip-10-0-129-137.ec2.internal
20:58:27	kserve-ci-e2e-test		message-dumper-predictor-6575bcfdfd-9c4gt	Scheduled	Successfully assigned kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9c4gt to ip-10-0-129-25.ec2.internal
21:07:39	kserve-ci-e2e-test		isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4 to ip-10-0-129-137.ec2.internal
21:03:38	kserve-ci-e2e-test		isvc-autogluon-v2-predictor-67b849d456-lth68	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-lth68 to ip-10-0-129-137.ec2.internal
20:58:47	kserve-ci-e2e-test		isvc-logger-predictor-5b7645fdfc-phwqx	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-logger-predictor-5b7645fdfc-phwqx to ip-10-0-129-137.ec2.internal
21:00:23	kserve-ci-e2e-test		isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs to ip-10-0-129-137.ec2.internal
20:55:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7595457646-gz6j8	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine
20:55:17	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
20:55:17	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-predictor-7595457646-gz6j8	AddedInterface	Add eth0 [10.132.0.36/23] from ovn-kubernetes
20:55:17	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher": the object has been modified; please apply your changes to the latest version and try again
20:55:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7595457646-gz6j8	Created	Created container: storage-initializer
20:55:17	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7595457646-gz6j8	Started	Started container storage-initializer
20:55:17	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-predictor-7595457646 from 0 to 1
20:55:17	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-predictor-7595457646	SuccessfulCreate	Created pod: isvc-sklearn-batcher-predictor-7595457646-gz6j8
20:55:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7595457646-gz6j8	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1683"
20:55:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7595457646-gz6j8	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
20:55:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7595457646-gz6j8	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1683" in 14.414s (14.414s including waiting). Image size: 1555648931 bytes.
20:55:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7595457646-gz6j8	Created	Created container: kserve-container
20:55:36	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7595457646-gz6j8	Started	Started container kserve-container
20:55:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7595457646-gz6j8	Pulling	Pulling image "quay.io/opendatahub/kserve-agent@sha256:eb91db6613fb4bde95991e8e02d444548e2e6e23bd5e4e0a11c1f350f3203183"
20:55:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7595457646-gz6j8	Started	Started container kube-rbac-proxy
20:55:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7595457646-gz6j8	Created	Created container: kube-rbac-proxy
20:55:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7595457646-gz6j8	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.445s (2.445s including waiting). Image size: 211946088 bytes.
20:55:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7595457646-gz6j8	Created	Created container: agent
20:55:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7595457646-gz6j8	Started	Started container agent
20:55:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7595457646-gz6j8	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:eb91db6613fb4bde95991e8e02d444548e2e6e23bd5e4e0a11c1f350f3203183" in 3.723s (3.723s including waiting). Image size: 239994455 bytes.
20:56:17 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:56:17 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:56:56	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	InferenceServiceReady	InferenceService [isvc-sklearn-batcher] is Ready
20:56:56 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:57:02 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:57:02	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-batcher-custom-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-batcher-custom-predictor-69d97fbc57 from 0 to 1
20:57:02	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-batcher-custom": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-batcher-custom": the object has been modified; please apply your changes to the latest version and try again
20:57:02	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-batcher-custom-predictor-69d97fbc57	SuccessfulCreate	Created pod: isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj
20:57:02 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:57:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7595457646-gz6j8	Killing	Stopping container kube-rbac-proxy
20:57:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7595457646-gz6j8	Killing	Stopping container agent
20:57:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7595457646-gz6j8	Killing	Stopping container kserve-container
20:57:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj	Started	Started container storage-initializer
20:57:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj	Created	Created container: storage-initializer
20:57:03	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine
20:57:03	kserve-ci-e2e-test	multus	isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj	AddedInterface	Add eth0 [10.132.0.37/23] from ovn-kubernetes
20:57:06 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7595457646-gz6j8	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
20:57:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:57:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj	Created	Created container: kube-rbac-proxy
20:57:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:eb91db6613fb4bde95991e8e02d444548e2e6e23bd5e4e0a11c1f350f3203183" already present on machine
20:57:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1683" already present on machine
20:57:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj	Created	Created container: kserve-container
20:57:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj	Started	Started container kserve-container
20:57:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj	Created	Created container: agent
20:57:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj	Started	Started container agent
20:57:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj	Started	Started container kube-rbac-proxy
20:57:14 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7595457646-gz6j8	Unhealthy	Readiness probe failed: Get "https://10.132.0.36:8643/healthz": dial tcp 10.132.0.36:8643: connect: connection refused
20:57:16 (x12)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-predictor-7595457646-gz6j8	Unhealthy	Readiness probe failed: dial tcp 10.132.0.36:8080: connect: connection refused
20:57:47 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:57:47 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:58:17 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:58:17 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-batcher-custom-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
20:58:21	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	InferenceServiceReady	InferenceService [isvc-sklearn-batcher-custom] is Ready
20:58:21 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-batcher-custom	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:58:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj	Killing	Stopping container kube-rbac-proxy
20:58:27	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	UpdateFailed	Failed to update status for InferenceService "message-dumper": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper": the object has been modified; please apply your changes to the latest version and try again
20:58:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj	Killing	Stopping container kserve-container
20:58:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj	Killing	Stopping container agent
20:58:27	kserve-ci-e2e-test	deployment-controller	message-dumper-predictor	ScalingReplicaSet	Scaled up replica set message-dumper-predictor-6575bcfdfd from 0 to 1
20:58:27	kserve-ci-e2e-test	replicaset-controller	message-dumper-predictor-6575bcfdfd	SuccessfulCreate	Created pod: message-dumper-predictor-6575bcfdfd-9c4gt
20:58:28	kserve-ci-e2e-test	multus	message-dumper-predictor-6575bcfdfd-9c4gt	AddedInterface	Add eth0 [10.133.0.28/23] from ovn-kubernetes
20:58:28	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-9c4gt	Pulling	Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display"
20:58:29	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-9c4gt	Pulled	Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.127s (1.127s including waiting). Image size: 14813193 bytes.
20:58:29	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-9c4gt	Created	Created container: kserve-container
20:58:29	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-9c4gt	Started	Started container kserve-container
20:58:29	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-9c4gt	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
20:58:31	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-9c4gt	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.24s (2.24s including waiting). Image size: 211946088 bytes.
20:58:31	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-9c4gt	Created	Created container: kube-rbac-proxy
20:58:31	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-9c4gt	Started	Started container kube-rbac-proxy
20:58:38 (x7)	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:58:38	kserve-ci-e2e-test	v1beta1Controllers	message-dumper	InferenceServiceReady	InferenceService [message-dumper] is Ready
20:58:39 (x3)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj	Unhealthy	Readiness probe failed: Get "https://10.132.0.37:8643/healthz": dial tcp 10.132.0.37:8643: connect: connection refused
20:58:41 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj	Unhealthy	Readiness probe failed: dial tcp 10.132.0.37:5000: connect: connection refused
20:58:41 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-batcher-custom-predictor-69d97fbc57-dz6jj	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
20:58:47	kserve-ci-e2e-test	deployment-controller	isvc-logger-predictor	ScalingReplicaSet	Scaled up replica set isvc-logger-predictor-5b7645fdfc from 0 to 1
20:58:47	kserve-ci-e2e-test	replicaset-controller	isvc-logger-predictor-5b7645fdfc	SuccessfulCreate	Created pod: isvc-logger-predictor-5b7645fdfc-phwqx
20:58:47	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	UpdateFailed	Failed to update status for InferenceService "isvc-logger": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger": the object has been modified; please apply your changes to the latest version and try again
20:58:48	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5b7645fdfc-phwqx	Created	Created container: storage-initializer
20:58:48	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5b7645fdfc-phwqx	Started	Started container storage-initializer
20:58:48	kserve-ci-e2e-test	multus	isvc-logger-predictor-5b7645fdfc-phwqx	AddedInterface	Add eth0 [10.132.0.38/23] from ovn-kubernetes
20:58:48	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5b7645fdfc-phwqx	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine
20:58:52	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5b7645fdfc-phwqx	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
20:58:52	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5b7645fdfc-phwqx	Created	Created container: kube-rbac-proxy
20:58:52	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5b7645fdfc-phwqx	Created	Created container: agent
20:58:52	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5b7645fdfc-phwqx	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1683" already present on machine
20:58:52	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5b7645fdfc-phwqx	Started	Started container agent
20:58:52	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5b7645fdfc-phwqx	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:eb91db6613fb4bde95991e8e02d444548e2e6e23bd5e4e0a11c1f350f3203183" already present on machine
20:58:52	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5b7645fdfc-phwqx	Started	Started container kube-rbac-proxy
20:58:52	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5b7645fdfc-phwqx	Created	Created container: kserve-container
20:58:52	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5b7645fdfc-phwqx	Started	Started container kserve-container
20:58:57 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:58:57 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:59:32 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:59:32 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:00:10	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	InferenceServiceReady	InferenceService [isvc-logger] is Ready
21:00:10 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:00:17 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:00:17 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:00:23	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-9c4gt	Killing	Stopping container kube-rbac-proxy
21:00:23	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v1": the object has been modified; please apply your changes to the latest version and try again
21:00:23	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v1-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v1-predictor-7bbbbd79cd from 0 to 1
21:00:23	kserve-ci-e2e-test	kubelet	message-dumper-predictor-6575bcfdfd-9c4gt	Killing	Stopping container kserve-container
21:00:23	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5b7645fdfc-phwqx	Killing	Stopping container agent
21:00:23	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v1-predictor-7bbbbd79cd	SuccessfulCreate	Created pod: isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs
21:00:23	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5b7645fdfc-phwqx	Killing	Stopping container kube-rbac-proxy
21:00:23	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5b7645fdfc-phwqx	Killing	Stopping container kserve-container
21:00:23	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v1": the object has been modified; please apply your changes to the latest version and try again
21:00:23	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-autogluon-v1-predictor-serving-cert" not found
21:00:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs	Started	Started container storage-initializer
21:00:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs	Created	Created container: storage-initializer
21:00:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine
21:00:24	kserve-ci-e2e-test	multus	isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs	AddedInterface	Add eth0 [10.132.0.39/23] from ovn-kubernetes
21:00:30 (x10)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5b7645fdfc-phwqx	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
21:00:30 (x11)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5b7645fdfc-phwqx	Unhealthy	Readiness probe failed: dial tcp 10.132.0.38:8080: connect: connection refused
21:00:40 (x4)	kserve-ci-e2e-test	kubelet	isvc-logger-predictor-5b7645fdfc-phwqx	Unhealthy	Readiness probe failed: Get "https://10.132.0.38:8643/healthz": dial tcp 10.132.0.38:8643: connect: connection refused
21:00:44	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs	Pulling	Pulling image "kserve/autogluonserver:latest"
21:03:05	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs	Pulled	Successfully pulled image "kserve/autogluonserver:latest" in 2m20.612s (2m20.612s including waiting). Image size: 8939170586 bytes.
21:03:05	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs	Started	Started container kube-rbac-proxy
21:03:05	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs	Created	Created container: kube-rbac-proxy
21:03:05	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:03:05	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs	Started	Started container kserve-container
21:03:05	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs	Created	Created container: kserve-container
21:03:08 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:03:08 (x11)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:03:23	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	InferenceServiceReady	InferenceService [isvc-autogluon-v1] is Ready
21:03:23 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:03:23	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:03:23	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:03:33	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs	Killing	Stopping container kube-rbac-proxy
21:03:33 (x4)	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs	Unhealthy	Readiness probe failed: dial tcp 10.132.0.39:8080: connect: connection refused
21:03:33	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs	Killing	Stopping container kserve-container
21:03:33	kserve-ci-e2e-test	kubelet	isvc-autogluon-v1-predictor-7bbbbd79cd-5m6gs	Unhealthy	Readiness probe failed: Get "https://10.132.0.39:8643/healthz": dial tcp 10.132.0.39:8643: connect: connection refused
21:03:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2": the object has been modified; please apply your changes to the latest version and try again
21:03:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2": the object has been modified; please apply your changes to the latest version and try again
21:03:38	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-predictor-67b849d456 from 0 to 1
21:03:38	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-predictor-67b849d456	SuccessfulCreate	Created pod: isvc-autogluon-v2-predictor-67b849d456-lth68
21:03:39	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-67b849d456-lth68	Started	Started container storage-initializer
21:03:39	kserve-ci-e2e-test	multus	isvc-autogluon-v2-predictor-67b849d456-lth68	AddedInterface	Add eth0 [10.132.0.40/23] from ovn-kubernetes
21:03:39	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-67b849d456-lth68	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine
21:03:39	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-67b849d456-lth68	Created	Created container: storage-initializer
21:04:08	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-67b849d456-lth68	Started	Started container kube-rbac-proxy
21:04:08	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-67b849d456-lth68	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
21:04:08	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-67b849d456-lth68	Created	Created container: kube-rbac-proxy
21:04:08	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-67b849d456-lth68	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:04:08	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-67b849d456-lth68	Started	Started container kserve-container
21:04:08	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-67b849d456-lth68	Created	Created container: kserve-container
21:04:38 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:04:38 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:05:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	InferenceServiceReady	InferenceService [isvc-autogluon-v2] is Ready
21:05:38 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:05:38 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:05:38 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:05:48	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-67b849d456-lth68	Killing	Stopping container kserve-container
21:05:48	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-predictor-67b849d456-lth68	Killing	Stopping container kube-rbac-proxy
21:05:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-variants": the object has been modified; please apply your changes to the latest version and try again
21:05:53	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-variants-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-variants-predictor-c6767c94c from 0 to 1
21:05:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2-variants": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-variants": the object has been modified; please apply your changes to the latest version and try again
21:05:53	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-variants-predictor-c6767c94c	SuccessfulCreate	Created pod: isvc-autogluon-v2-variants-predictor-c6767c94c-6d854
21:05:54	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-c6767c94c-6d854	Created	Created container: storage-initializer
21:05:54	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-c6767c94c-6d854	Started	Started container storage-initializer
21:05:54	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-c6767c94c-6d854	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine
21:05:54	kserve-ci-e2e-test	multus	isvc-autogluon-v2-variants-predictor-c6767c94c-6d854	AddedInterface	Add eth0 [10.132.0.41/23] from ovn-kubernetes
21:05:59	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-c6767c94c-6d854	Created	Created container: kube-rbac-proxy
21:05:59	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-c6767c94c-6d854	Started	Started container kube-rbac-proxy
21:05:59	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-c6767c94c-6d854	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:05:59	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-c6767c94c-6d854	Started	Started container kserve-container
21:05:59	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-c6767c94c-6d854	Created	Created container: kserve-container
21:05:59	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-c6767c94c-6d854	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
21:06:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:06:23 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:07:23 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:07:23 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-variants-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:07:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	InferenceServiceReady	InferenceService [isvc-autogluon-v2-variants] is Ready
21:07:31 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-variants	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:07:33	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-c6767c94c-6d854	Killing	Stopping container kserve-container
21:07:33	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-variants-predictor-c6767c94c-6d854	Killing	Stopping container kube-rbac-proxy
21:07:39	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-v2-noslash-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-v2-noslash-predictor-9dcfbbb84 from 0 to 1
21:07:39	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4	Created	Created container: storage-initializer
21:07:39 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-noslash": the object has been modified; please apply your changes to the latest version and try again
21:07:39 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-v2-noslash": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-v2-noslash": the object has been modified; please apply your changes to the latest version and try again
21:07:39	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4	Started	Started container storage-initializer
21:07:39	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine
21:07:39	kserve-ci-e2e-test	multus	isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4	AddedInterface	Add eth0 [10.132.0.42/23] from ovn-kubernetes
21:07:39	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-v2-noslash-predictor-9dcfbbb84	SuccessfulCreate	Created pod: isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4
21:07:44	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:07:44	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4	Started	Started container kube-rbac-proxy
21:07:44	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4	Created	Created container: kube-rbac-proxy
21:07:44	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4	Started	Started container kserve-container
21:07:44	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4	Created	Created container: kserve-container
21:07:44	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
21:07:54	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:07:54	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:09:09 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:09:09 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-v2-noslash-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:09:14 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:09:14	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-v2-noslash	InferenceServiceReady	InferenceService [isvc-autogluon-v2-noslash] is Ready
21:09:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4	Killing	Stopping container kube-rbac-proxy
21:09:19	kserve-ci-e2e-test	kubelet	isvc-autogluon-v2-noslash-predictor-9dcfbbb84-4k2g4	Killing	Stopping container kserve-container
21:09:24	kserve-ci-e2e-test	replicaset-controller	isvc-autogluon-ts-v1-predictor-89d67dbf7	SuccessfulCreate	Created pod: isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh
21:09:24	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1": the object has been modified; please apply your changes to the latest version and try again
21:09:24	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	UpdateFailed	Failed to update status for InferenceService "isvc-autogluon-ts-v1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-autogluon-ts-v1": the object has been modified; please apply your changes to the latest version and try again
21:09:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh	Started	Started container storage-initializer
21:09:24	kserve-ci-e2e-test	multus	isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh	AddedInterface	Add eth0 [10.132.0.43/23] from ovn-kubernetes
21:09:24	kserve-ci-e2e-test	deployment-controller	isvc-autogluon-ts-v1-predictor	ScalingReplicaSet	Scaled up replica set isvc-autogluon-ts-v1-predictor-89d67dbf7 from 0 to 1
21:09:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh	Created	Created container: storage-initializer
21:09:24	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine
21:09:35	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh	Started	Started container kube-rbac-proxy
21:09:35	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh	Created	Created container: kube-rbac-proxy
21:09:35	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:09:35	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh	Started	Started container kserve-container
21:09:35	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh	Created	Created container: kserve-container
21:09:35	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh	Pulled	Container image "kserve/autogluonserver:latest" already present on machine
21:09:39	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:09:39	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:09:43 (x3)	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh	Unhealthy	Readiness probe failed: dial tcp 10.132.0.43:8080: connect: connection refused
21:09:53 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:09:53	kserve-ci-e2e-test	v1beta1Controllers	isvc-autogluon-ts-v1	InferenceServiceReady	InferenceService [isvc-autogluon-ts-v1] is Ready
21:09:54	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:09:54	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh	Killing	Stopping container kserve-container
21:09:54	kserve-ci-e2e-test	kubelet	isvc-autogluon-ts-v1-predictor-89d67dbf7-7gmvh	Killing	Stopping container kube-rbac-proxy
21:09:54	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-autogluon-ts-v1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)