Events

Time	Namespace	Component	RelatedObject	Reason	Message
18:23:35	kserve-ci-e2e-test		isvc-primary-72dc06-predictor-6cb65cb8db-9fztt	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-primary-72dc06-predictor-6cb65cb8db-9fztt to ip-10-0-132-145.ec2.internal
18:15:04	kserve-ci-e2e-test		isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf to ip-10-0-132-145.ec2.internal
18:25:39	kserve-ci-e2e-test		raw-sklearn-20421-predictor-58f54f4cf9-czsvq	Scheduled	Successfully assigned kserve-ci-e2e-test/raw-sklearn-20421-predictor-58f54f4cf9-czsvq to ip-10-0-132-145.ec2.internal
18:16:54	kserve-ci-e2e-test		isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p to ip-10-0-132-145.ec2.internal
18:13:18	kserve-ci-e2e-test		isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5 to ip-10-0-132-145.ec2.internal
18:25:28	kserve-ci-e2e-test		isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb to ip-10-0-132-145.ec2.internal
18:20:40	kserve-ci-e2e-test		isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj to ip-10-0-132-145.ec2.internal
18:16:44	kserve-ci-e2e-test		model-chainer-raw-ebf1b-8479fd844c-d2m5f	Scheduled	Successfully assigned kserve-ci-e2e-test/model-chainer-raw-ebf1b-8479fd844c-d2m5f to ip-10-0-136-131.ec2.internal
18:16:54	kserve-ci-e2e-test		isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj to ip-10-0-132-145.ec2.internal
18:25:05	kserve-ci-e2e-test		isvc-secondary-72dc06-predictor-6477d9654f-jlnkh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-secondary-72dc06-predictor-6477d9654f-jlnkh to ip-10-0-132-145.ec2.internal
18:18:44	kserve-ci-e2e-test		message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7	Scheduled	Successfully assigned kserve-ci-e2e-test/message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7 to ip-10-0-132-145.ec2.internal
18:19:04	kserve-ci-e2e-test		isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc to ip-10-0-132-145.ec2.internal
18:15:04	kserve-ci-e2e-test		isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc to ip-10-0-132-145.ec2.internal
18:18:34	kserve-ci-e2e-test		model-chainer-raw-hpa-c9b93-67b958dfc9-978rt	Scheduled	Successfully assigned kserve-ci-e2e-test/model-chainer-raw-hpa-c9b93-67b958dfc9-978rt to ip-10-0-136-131.ec2.internal
18:27:14	kserve-ci-e2e-test		raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6	Scheduled	Successfully assigned kserve-ci-e2e-test/raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6 to ip-10-0-132-145.ec2.internal
18:13:18	kserve-ci-e2e-test	deployment-controller	isvc-raw-sklearn-batcher-a980c-predictor	ScalingReplicaSet	Scaled up replica set isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5 from 0 to 1
18:13:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-raw-sklearn-batcher-a980c	UpdateFailed	Failed to update status for InferenceService "isvc-raw-sklearn-batcher-a980c": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-a980c": the object has been modified; please apply your changes to the latest version and try again
18:13:18	kserve-ci-e2e-test	v1beta1Controllers	isvc-raw-sklearn-batcher-a980c	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-a980c": the object has been modified; please apply your changes to the latest version and try again
18:13:18	kserve-ci-e2e-test	replicaset-controller	isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5	SuccessfulCreate	Created pod: isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5
18:13:19	kserve-ci-e2e-test	multus	isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5	AddedInterface	Add eth0 [10.134.0.26/23] from ovn-kubernetes
18:13:19	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-raw-sklearn-batcher-a980c-predictor-serving-cert" not found
18:13:20	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5	Created	Created container: storage-initializer
18:13:20	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d9a53871119e5163dc10506afed26a0cdd4198fd2315ea323b1e9fe09949b475" already present on machine
18:13:20	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5	Started	Started container storage-initializer
18:13:23	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1684"
18:13:38	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
18:13:38	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1684" in 14.982s (14.982s including waiting). Image size: 1559186858 bytes.
18:13:38	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5	Started	Started container kserve-container
18:13:38	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5	Created	Created container: kserve-container
18:13:41	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5	Pulling	Pulling image "quay.io/opendatahub/kserve-agent@sha256:2fd41eb0f45027343da0a3e7ab10342a4e3f1970b88cf54fcc671e20be2ec985"
18:13:41	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.783s (2.783s including waiting). Image size: 211946088 bytes.
18:13:41	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5	Created	Created container: kube-rbac-proxy
18:13:41	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5	Started	Started container kube-rbac-proxy
18:13:44	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:2fd41eb0f45027343da0a3e7ab10342a4e3f1970b88cf54fcc671e20be2ec985" in 3.111s (3.111s including waiting). Image size: 239994467 bytes.
18:13:45	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5	Created	Created container: agent
18:13:45	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5	Started	Started container agent
18:14:19 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-raw-sklearn-batcher-a980c-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:14:19 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-raw-sklearn-batcher-a980c-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:14:58 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-raw-sklearn-batcher-a980c	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:14:58	kserve-ci-e2e-test	v1beta1Controllers	isvc-raw-sklearn-batcher-a980c	InferenceServiceReady	InferenceService [isvc-raw-sklearn-batcher-a980c] is Ready
18:15:04	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5	Killing	Stopping container kube-rbac-proxy
18:15:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d9a53871119e5163dc10506afed26a0cdd4198fd2315ea323b1e9fe09949b475" already present on machine
18:15:04	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5	Killing	Stopping container kserve-container
18:15:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-graph-raw-ebf1b	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-graph-raw-ebf1b": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-graph-raw-ebf1b": the object has been modified; please apply your changes to the latest version and try again
18:15:04 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-raw-sklearn-batcher-a980c-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:15:04	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5	Killing	Stopping container agent
18:15:04 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-raw-sklearn-batcher-a980c-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:15:04	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf	SuccessfulCreate	Created pod: isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc
18:15:04	kserve-ci-e2e-test	multus	isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf	AddedInterface	Add eth0 [10.134.0.27/23] from ovn-kubernetes
18:15:04	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-graph-raw-ebf1b-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf from 0 to 1
18:15:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf	Created	Created container: storage-initializer
18:15:04	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf	Started	Started container storage-initializer
18:15:04	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495	SuccessfulCreate	Created pod: isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf
18:15:04	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-graph-raw-ebf1b-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495 from 0 to 1
18:15:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-raw-ebf1b	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-graph-raw-ebf1b": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-ebf1b": the object has been modified; please apply your changes to the latest version and try again
18:15:04	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc	Started	Started container storage-initializer
18:15:04	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc	Created	Created container: storage-initializer
18:15:04	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d9a53871119e5163dc10506afed26a0cdd4198fd2315ea323b1e9fe09949b475" already present on machine
18:15:04	kserve-ci-e2e-test	multus	isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc	AddedInterface	Add eth0 [10.134.0.28/23] from ovn-kubernetes
18:15:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf	Started	Started container kube-rbac-proxy
18:15:08 (x11)	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5	Unhealthy	Readiness probe failed: dial tcp 10.134.0.26:8080: connect: connection refused
18:15:08 (x10)	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
18:15:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1684" already present on machine
18:15:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf	Created	Created container: kserve-container
18:15:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf	Started	Started container kserve-container
18:15:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:15:08	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc	Pulling	Pulling image "kserve/xgbserver:latest"
18:15:08	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf	Created	Created container: kube-rbac-proxy
18:15:16 (x3)	kserve-ci-e2e-test	kubelet	isvc-raw-sklearn-batcher-a980c-predictor-7db9b49dc5-xxph5	Unhealthy	Readiness probe failed: Get "https://10.134.0.26:8643/healthz": dial tcp 10.134.0.26:8643: connect: connection refused
18:15:28	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 20.568s (20.568s including waiting). Image size: 1305293052 bytes.
18:15:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc	Created	Created container: kserve-container
18:15:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc	Started	Started container kube-rbac-proxy
18:15:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc	Created	Created container: kube-rbac-proxy
18:15:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc	Started	Started container kserve-container
18:15:29	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:15:49 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-raw-ebf1b-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:15:49 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-raw-ebf1b-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:16:04 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-raw-ebf1b-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:16:04 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-raw-ebf1b-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:16:27	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-raw-ebf1b	InferenceServiceReady	InferenceService [isvc-sklearn-graph-raw-ebf1b] is Ready
18:16:28 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-raw-ebf1b	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:16:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-graph-raw-ebf1b	InferenceServiceReady	InferenceService [isvc-xgboost-graph-raw-ebf1b] is Ready
18:16:35 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-graph-raw-ebf1b	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:16:44	kserve-ci-e2e-test	InferenceGraphController	model-chainer-raw-ebf1b	InferenceGraphReady	InferenceGraph [model-chainer-raw-ebf1b] is Ready
18:16:44	kserve-ci-e2e-test	multus	model-chainer-raw-ebf1b-8479fd844c-d2m5f	AddedInterface	Add eth0 [10.132.0.18/23] from ovn-kubernetes
18:16:44	kserve-ci-e2e-test	replicaset-controller	model-chainer-raw-ebf1b-8479fd844c	SuccessfulCreate	Created pod: model-chainer-raw-ebf1b-8479fd844c-d2m5f
18:16:44	kserve-ci-e2e-test	deployment-controller	model-chainer-raw-ebf1b	ScalingReplicaSet	Scaled up replica set model-chainer-raw-ebf1b-8479fd844c from 0 to 1
18:16:44	kserve-ci-e2e-test	kubelet	model-chainer-raw-ebf1b-8479fd844c-d2m5f	Pulling	Pulling image "quay.io/opendatahub/kserve-router@sha256:4d6fcc0e5d3b23dc1e66af36b5e4c4d4c9974f1880f849ee4e4508e9f5fd89ff"
18:16:47	kserve-ci-e2e-test	kubelet	model-chainer-raw-ebf1b-8479fd844c-d2m5f	Started	Started container model-chainer-raw-ebf1b
18:16:47	kserve-ci-e2e-test	kubelet	model-chainer-raw-ebf1b-8479fd844c-d2m5f	Created	Created container: model-chainer-raw-ebf1b
18:16:47	kserve-ci-e2e-test	kubelet	model-chainer-raw-ebf1b-8479fd844c-d2m5f	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:4d6fcc0e5d3b23dc1e66af36b5e4c4d4c9974f1880f849ee4e4508e9f5fd89ff" in 2.448s (2.448s including waiting). Image size: 218070118 bytes.
18:16:49 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-raw-ebf1b-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:16:49 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-raw-ebf1b-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:16:49 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-raw-ebf1b-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:16:49 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-raw-ebf1b-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:16:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d9a53871119e5163dc10506afed26a0cdd4198fd2315ea323b1e9fe09949b475" already present on machine
18:16:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-graph-raw-hpa-c9b93	UpdateFailed	Failed to update status for InferenceService "isvc-xgboost-graph-raw-hpa-c9b93": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-graph-raw-hpa-c9b93": the object has been modified; please apply your changes to the latest version and try again
18:16:54	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-raw-hpa-c9b93	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-graph-raw-hpa-c9b93": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-hpa-c9b93": the object has been modified; please apply your changes to the latest version and try again
18:16:54	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-graph-raw-hpa-c9b93-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f from 0 to 1
18:16:54	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f	SuccessfulCreate	Created pod: isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj
18:16:54	kserve-ci-e2e-test	multus	isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p	AddedInterface	Add eth0 [10.134.0.30/23] from ovn-kubernetes
18:16:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf	Killing	Stopping container kserve-container
18:16:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj	Started	Started container storage-initializer
18:16:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf	Killing	Stopping container kube-rbac-proxy
18:16:54	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc	Killing	Stopping container kube-rbac-proxy
18:16:54	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc	Killing	Stopping container kserve-container
18:16:54	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf	SuccessfulCreate	Created pod: isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p
18:16:54	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj	Created	Created container: storage-initializer
18:16:54	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-graph-raw-hpa-c9b93-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf from 0 to 1
18:16:54	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d9a53871119e5163dc10506afed26a0cdd4198fd2315ea323b1e9fe09949b475" already present on machine
18:16:54	kserve-ci-e2e-test	multus	isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj	AddedInterface	Add eth0 [10.134.0.29/23] from ovn-kubernetes
18:16:54	kserve-ci-e2e-test	kubelet	model-chainer-raw-ebf1b-8479fd844c-d2m5f	Killing	Stopping container model-chainer-raw-ebf1b
18:16:55	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p	Started	Started container storage-initializer
18:16:55	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p	Created	Created container: storage-initializer
18:16:55 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc	Unhealthy	Readiness probe failed: dial tcp 10.134.0.28:8080: connect: connection refused
18:16:55	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-ebf1b-predictor-56996dd5bf-s96kc	Unhealthy	Readiness probe failed: Get "https://10.134.0.28:8643/healthz": dial tcp 10.134.0.28:8643: connect: connection refused
18:16:56	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf	Unhealthy	Readiness probe failed: Get "https://10.134.0.27:8643/healthz": dial tcp 10.134.0.27:8643: connect: connection refused
18:16:57 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-ebf1b-predictor-85b9f95495-z84rf	Unhealthy	Readiness probe failed: dial tcp 10.134.0.27:8080: connect: connection refused
18:16:58	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p	Pulled	Container image "kserve/xgbserver:latest" already present on machine
18:16:58	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1684" already present on machine
18:16:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj	Created	Created container: kube-rbac-proxy
18:16:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj	Created	Created container: kserve-container
18:16:59	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p	Started	Started container kube-rbac-proxy
18:16:59	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p	Created	Created container: kube-rbac-proxy
18:16:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj	Started	Started container kube-rbac-proxy
18:16:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:16:59	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj	Started	Started container kserve-container
18:16:59	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p	Created	Created container: kserve-container
18:16:59	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p	Started	Started container kserve-container
18:16:59	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:17:23 (x6)	kserve-ci-e2e-test	kubelet	model-chainer-raw-ebf1b-8479fd844c-d2m5f	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
18:17:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-raw-hpa-c9b93-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:17:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-raw-hpa-c9b93-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:17:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-raw-hpa-c9b93-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:17:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-raw-hpa-c9b93-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:18:06 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-graph-raw-hpa-c9b93	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:18:06	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-graph-raw-hpa-c9b93	InferenceServiceReady	InferenceService [isvc-xgboost-graph-raw-hpa-c9b93] is Ready
18:18:17 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-raw-hpa-c9b93	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:18:17	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-raw-hpa-c9b93	InferenceServiceReady	InferenceService [isvc-sklearn-graph-raw-hpa-c9b93] is Ready
18:18:34	kserve-ci-e2e-test	replicaset-controller	model-chainer-raw-hpa-c9b93-67b958dfc9	SuccessfulCreate	Created pod: model-chainer-raw-hpa-c9b93-67b958dfc9-978rt
18:18:34	kserve-ci-e2e-test	deployment-controller	model-chainer-raw-hpa-c9b93	ScalingReplicaSet	Scaled up replica set model-chainer-raw-hpa-c9b93-67b958dfc9 from 0 to 1
18:18:34	kserve-ci-e2e-test	InferenceGraphController	model-chainer-raw-hpa-c9b93	InferenceGraphReady	InferenceGraph [model-chainer-raw-hpa-c9b93] is Ready
18:18:34	kserve-ci-e2e-test	kubelet	model-chainer-raw-hpa-c9b93-67b958dfc9-978rt	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-raw-hpa-c9b93-serving-cert" not found
18:18:35	kserve-ci-e2e-test	multus	model-chainer-raw-hpa-c9b93-67b958dfc9-978rt	AddedInterface	Add eth0 [10.132.0.19/23] from ovn-kubernetes
18:18:35	kserve-ci-e2e-test	kubelet	model-chainer-raw-hpa-c9b93-67b958dfc9-978rt	Started	Started container model-chainer-raw-hpa-c9b93
18:18:35	kserve-ci-e2e-test	kubelet	model-chainer-raw-hpa-c9b93-67b958dfc9-978rt	Created	Created container: model-chainer-raw-hpa-c9b93
18:18:35	kserve-ci-e2e-test	kubelet	model-chainer-raw-hpa-c9b93-67b958dfc9-978rt	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:4d6fcc0e5d3b23dc1e66af36b5e4c4d4c9974f1880f849ee4e4508e9f5fd89ff" already present on machine
18:18:39 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-raw-hpa-c9b93-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:18:39 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-raw-hpa-c9b93-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:18:39 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-raw-hpa-c9b93-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:18:39 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-raw-hpa-c9b93-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:18:44	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p	Killing	Stopping container kserve-container
18:18:44	kserve-ci-e2e-test	v1beta1Controllers	message-dumper-raw-dbad7	UpdateFailed	Failed to update status for InferenceService "message-dumper-raw-dbad7": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-dbad7": the object has been modified; please apply your changes to the latest version and try again
18:18:44	kserve-ci-e2e-test	replicaset-controller	message-dumper-raw-dbad7-predictor-94b4c95bd	SuccessfulCreate	Created pod: message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7
18:18:44	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p	Killing	Stopping container kube-rbac-proxy
18:18:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj	Killing	Stopping container kube-rbac-proxy
18:18:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj	Killing	Stopping container kserve-container
18:18:44	kserve-ci-e2e-test	deployment-controller	message-dumper-raw-dbad7-predictor	ScalingReplicaSet	Scaled up replica set message-dumper-raw-dbad7-predictor-94b4c95bd from 0 to 1
18:18:44	kserve-ci-e2e-test	kubelet	message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "message-dumper-raw-dbad7-predictor-serving-cert" not found
18:18:44	kserve-ci-e2e-test	kubelet	model-chainer-raw-hpa-c9b93-67b958dfc9-978rt	Killing	Stopping container model-chainer-raw-hpa-c9b93
18:18:45	kserve-ci-e2e-test	kubelet	message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7	Pulling	Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display"
18:18:45	kserve-ci-e2e-test	multus	message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7	AddedInterface	Add eth0 [10.134.0.31/23] from ovn-kubernetes
18:18:46	kserve-ci-e2e-test	kubelet	message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7	Pulled	Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.202s (1.202s including waiting). Image size: 14813193 bytes.
18:18:46 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p	Unhealthy	Readiness probe failed: dial tcp 10.134.0.30:8080: connect: connection refused
18:18:46	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-raw-hpa-c9b93-predictor-76bd69bcdf-zhn6p	Unhealthy	Readiness probe failed: Get "https://10.134.0.30:8643/healthz": dial tcp 10.134.0.30:8643: connect: connection refused
18:18:47	kserve-ci-e2e-test	kubelet	message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7	Created	Created container: kube-rbac-proxy
18:18:47	kserve-ci-e2e-test	kubelet	message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7	Created	Created container: kserve-container
18:18:47 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj	Unhealthy	Readiness probe failed: dial tcp 10.134.0.29:8080: connect: connection refused
18:18:47	kserve-ci-e2e-test	kubelet	message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7	Started	Started container kube-rbac-proxy
18:18:47	kserve-ci-e2e-test	kubelet	message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:18:47	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-raw-hpa-c9b93-predictor-7f4844db9f-z9hdj	Unhealthy	Readiness probe failed: Get "https://10.134.0.29:8643/healthz": dial tcp 10.134.0.29:8643: connect: connection refused
18:18:47	kserve-ci-e2e-test	kubelet	message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7	Started	Started container kserve-container
18:18:54 (x7)	kserve-ci-e2e-test	v1beta1Controllers	message-dumper-raw-dbad7	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:18:54	kserve-ci-e2e-test	v1beta1Controllers	message-dumper-raw-dbad7	InferenceServiceReady	InferenceService [message-dumper-raw-dbad7] is Ready
18:18:59	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-raw-dbad7-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:18:59	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-raw-dbad7-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:19:04	kserve-ci-e2e-test	replicaset-controller	isvc-logger-raw-dbad7-predictor-f57b9b598	SuccessfulCreate	Created pod: isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc
18:19:04	kserve-ci-e2e-test	kubelet	isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-logger-raw-dbad7-predictor-serving-cert" not found
18:19:04	kserve-ci-e2e-test	deployment-controller	isvc-logger-raw-dbad7-predictor	ScalingReplicaSet	Scaled up replica set isvc-logger-raw-dbad7-predictor-f57b9b598 from 0 to 1
18:19:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger-raw-dbad7	UpdateFailed	Failed to update status for InferenceService "isvc-logger-raw-dbad7": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-dbad7": the object has been modified; please apply your changes to the latest version and try again
18:19:05	kserve-ci-e2e-test	kubelet	isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d9a53871119e5163dc10506afed26a0cdd4198fd2315ea323b1e9fe09949b475" already present on machine
18:19:05	kserve-ci-e2e-test	multus	isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc	AddedInterface	Add eth0 [10.134.0.32/23] from ovn-kubernetes
18:19:05	kserve-ci-e2e-test	kubelet	isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc	Created	Created container: storage-initializer
18:19:05	kserve-ci-e2e-test	kubelet	isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc	Started	Started container storage-initializer
18:19:09	kserve-ci-e2e-test	kubelet	isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc	Started	Started container kube-rbac-proxy
18:19:09	kserve-ci-e2e-test	kubelet	isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:19:09	kserve-ci-e2e-test	kubelet	isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1684" already present on machine
18:19:09	kserve-ci-e2e-test	kubelet	isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc	Started	Started container kserve-container
18:19:09	kserve-ci-e2e-test	kubelet	isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc	Created	Created container: kserve-container
18:19:09	kserve-ci-e2e-test	kubelet	isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc	Started	Started container agent
18:19:09	kserve-ci-e2e-test	kubelet	isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc	Created	Created container: kube-rbac-proxy
18:19:09	kserve-ci-e2e-test	kubelet	isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc	Created	Created container: agent
18:19:09	kserve-ci-e2e-test	kubelet	isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc	Pulled	Container image "quay.io/opendatahub/kserve-agent@sha256:2fd41eb0f45027343da0a3e7ab10342a4e3f1970b88cf54fcc671e20be2ec985" already present on machine
18:19:11 (x6)	kserve-ci-e2e-test	kubelet	model-chainer-raw-hpa-c9b93-67b958dfc9-978rt	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
18:19:14	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-raw-dbad7-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:19:14	kserve-ci-e2e-test	horizontal-pod-autoscaler	message-dumper-raw-dbad7-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:19:49 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-raw-dbad7-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:19:49 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-raw-dbad7-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:20:24	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger-raw-dbad7	InferenceServiceReady	InferenceService [isvc-logger-raw-dbad7] is Ready
18:20:24 (x11)	kserve-ci-e2e-test	v1beta1Controllers	isvc-logger-raw-dbad7	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:20:34 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-raw-dbad7-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:20:34 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-logger-raw-dbad7-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:20:39	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-scale-raw-7a9f1-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45 from 0 to 1
18:20:40	kserve-ci-e2e-test	kubelet	isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc	Killing	Stopping container agent
18:20:40 (x2)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scale-raw-7a9f1	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-scale-raw-7a9f1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scale-raw-7a9f1": the object has been modified; please apply your changes to the latest version and try again
18:20:40	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scale-raw-7a9f1	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scale-raw-7a9f1": the object has been modified; please apply your changes to the latest version and try again
18:20:40	kserve-ci-e2e-test	kubelet	isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc	Killing	Stopping container kserve-container
18:20:40	kserve-ci-e2e-test	multus	isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj	AddedInterface	Add eth0 [10.134.0.33/23] from ovn-kubernetes
18:20:40	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45	SuccessfulCreate	Created pod: isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj
18:20:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d9a53871119e5163dc10506afed26a0cdd4198fd2315ea323b1e9fe09949b475" already present on machine
18:20:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj	Created	Created container: storage-initializer
18:20:40	kserve-ci-e2e-test	kubelet	message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7	Killing	Stopping container kube-rbac-proxy
18:20:40	kserve-ci-e2e-test	kubelet	isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc	Killing	Stopping container kube-rbac-proxy
18:20:40	kserve-ci-e2e-test	kubelet	message-dumper-raw-dbad7-predictor-94b4c95bd-mgtg7	Killing	Stopping container kserve-container
18:20:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj	Started	Started container storage-initializer
18:20:44 (x10)	kserve-ci-e2e-test	kubelet	isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
18:20:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj	Started	Started container kube-rbac-proxy
18:20:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1684" already present on machine
18:20:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj	Created	Created container: kserve-container
18:20:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj	Started	Started container kserve-container
18:20:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:20:45	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj	Created	Created container: kube-rbac-proxy
18:20:51 (x3)	kserve-ci-e2e-test	kubelet	isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc	Unhealthy	Readiness probe failed: Get "https://10.134.0.32:8643/healthz": dial tcp 10.134.0.32:8643: connect: connection refused
18:20:54 (x11)	kserve-ci-e2e-test	kubelet	isvc-logger-raw-dbad7-predictor-f57b9b598-6fzwc	Unhealthy	Readiness probe failed: dial tcp 10.134.0.32:8080: connect: connection refused
18:21:25 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-scale-raw-7a9f1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:21:25 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-scale-raw-7a9f1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:23:22 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scale-raw-7a9f1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:23:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-scale-raw-7a9f1	InferenceServiceReady	InferenceService [isvc-sklearn-scale-raw-7a9f1] is Ready
18:23:25 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-scale-raw-7a9f1-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:23:25 (x8)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-scale-raw-7a9f1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:23:30	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj	Killing	Stopping container kube-rbac-proxy
18:23:30	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj	Killing	Stopping container kserve-container
18:23:32 (x19)	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj	Unhealthy	Readiness probe failed: dial tcp 10.134.0.33:8080: connect: connection refused
18:23:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-72dc06	UpdateFailed	Failed to update status for InferenceService "isvc-primary-72dc06": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-72dc06": the object has been modified; please apply your changes to the latest version and try again
18:23:35	kserve-ci-e2e-test	multus	isvc-primary-72dc06-predictor-6cb65cb8db-9fztt	AddedInterface	Add eth0 [10.134.0.34/23] from ovn-kubernetes
18:23:35	kserve-ci-e2e-test	kubelet	isvc-primary-72dc06-predictor-6cb65cb8db-9fztt	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d9a53871119e5163dc10506afed26a0cdd4198fd2315ea323b1e9fe09949b475" already present on machine
18:23:35	kserve-ci-e2e-test	kubelet	isvc-primary-72dc06-predictor-6cb65cb8db-9fztt	Created	Created container: storage-initializer
18:23:35	kserve-ci-e2e-test	kubelet	isvc-primary-72dc06-predictor-6cb65cb8db-9fztt	Started	Started container storage-initializer
18:23:35	kserve-ci-e2e-test	deployment-controller	isvc-primary-72dc06-predictor	ScalingReplicaSet	Scaled up replica set isvc-primary-72dc06-predictor-6cb65cb8db from 0 to 1
18:23:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-72dc06	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-72dc06": the object has been modified; please apply your changes to the latest version and try again
18:23:35	kserve-ci-e2e-test	replicaset-controller	isvc-primary-72dc06-predictor-6cb65cb8db	SuccessfulCreate	Created pod: isvc-primary-72dc06-predictor-6cb65cb8db-9fztt
18:23:37 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-scale-raw-7a9f1-predictor-79655cbf45-zkcvj	Unhealthy	Readiness probe failed: Get "https://10.134.0.33:8643/healthz": dial tcp 10.134.0.33:8643: connect: connection refused
18:23:39	kserve-ci-e2e-test	kubelet	isvc-primary-72dc06-predictor-6cb65cb8db-9fztt	Started	Started container kserve-container
18:23:39	kserve-ci-e2e-test	kubelet	isvc-primary-72dc06-predictor-6cb65cb8db-9fztt	Started	Started container kube-rbac-proxy
18:23:39	kserve-ci-e2e-test	kubelet	isvc-primary-72dc06-predictor-6cb65cb8db-9fztt	Created	Created container: kube-rbac-proxy
18:23:39	kserve-ci-e2e-test	kubelet	isvc-primary-72dc06-predictor-6cb65cb8db-9fztt	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:23:39	kserve-ci-e2e-test	kubelet	isvc-primary-72dc06-predictor-6cb65cb8db-9fztt	Created	Created container: kserve-container
18:23:39	kserve-ci-e2e-test	kubelet	isvc-primary-72dc06-predictor-6cb65cb8db-9fztt	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1684" already present on machine
18:23:50	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-72dc06-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:23:50	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-72dc06-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:24:56	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-72dc06	InferenceServiceReady	InferenceService [isvc-primary-72dc06] is Ready
18:24:57 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-primary-72dc06	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:25:05	kserve-ci-e2e-test	deployment-controller	isvc-secondary-72dc06-predictor	ScalingReplicaSet	Scaled up replica set isvc-secondary-72dc06-predictor-6477d9654f from 0 to 1
18:25:05	kserve-ci-e2e-test	replicaset-controller	isvc-secondary-72dc06-predictor-6477d9654f	SuccessfulCreate	Created pod: isvc-secondary-72dc06-predictor-6477d9654f-jlnkh
18:25:05	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-72dc06	UpdateFailed	Failed to update status for InferenceService "isvc-secondary-72dc06": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-72dc06": the object has been modified; please apply your changes to the latest version and try again
18:25:05	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-72dc06	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-72dc06": the object has been modified; please apply your changes to the latest version and try again
18:25:06	kserve-ci-e2e-test	multus	isvc-secondary-72dc06-predictor-6477d9654f-jlnkh	AddedInterface	Add eth0 [10.134.0.35/23] from ovn-kubernetes
18:25:13 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-72dc06-predictor-6477d9654f-jlnkh	Started	Started container storage-initializer
18:25:13 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-72dc06-predictor-6477d9654f-jlnkh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d9a53871119e5163dc10506afed26a0cdd4198fd2315ea323b1e9fe09949b475" already present on machine
18:25:13 (x2)	kserve-ci-e2e-test	kubelet	isvc-secondary-72dc06-predictor-6477d9654f-jlnkh	Created	Created container: storage-initializer
18:25:19	kserve-ci-e2e-test	kubelet	isvc-secondary-72dc06-predictor-6477d9654f-jlnkh	BackOff	Back-off restarting failed container storage-initializer in pod isvc-secondary-72dc06-predictor-6477d9654f-jlnkh_kserve-ci-e2e-test(817dac3b-5c8a-4d5d-b0f3-febe0ade7c95)
18:25:19 (x12)	kserve-ci-e2e-test	v1beta1Controllers	isvc-secondary-72dc06	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:25:20 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-72dc06-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:25:20 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-primary-72dc06-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:25:20	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-72dc06-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:25:20	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-secondary-72dc06-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:25:23	kserve-ci-e2e-test	kubelet	isvc-primary-72dc06-predictor-6cb65cb8db-9fztt	Killing	Stopping container kube-rbac-proxy
18:25:23	kserve-ci-e2e-test	kubelet	isvc-primary-72dc06-predictor-6cb65cb8db-9fztt	Killing	Stopping container kserve-container
18:25:26	kserve-ci-e2e-test	kubelet	isvc-primary-72dc06-predictor-6cb65cb8db-9fztt	Unhealthy	Readiness probe failed: Get "https://10.134.0.34:8643/healthz": dial tcp 10.134.0.34:8643: connect: connection refused
18:25:26 (x10)	kserve-ci-e2e-test	kubelet	isvc-primary-72dc06-predictor-6cb65cb8db-9fztt	Unhealthy	Readiness probe failed: dial tcp 10.134.0.34:8080: connect: connection refused
18:25:28	kserve-ci-e2e-test	replicaset-controller	isvc-init-fail-b713a0-predictor-77df54fb5f	SuccessfulCreate	Created pod: isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb
18:25:28	kserve-ci-e2e-test	deployment-controller	isvc-init-fail-b713a0-predictor	ScalingReplicaSet	Scaled up replica set isvc-init-fail-b713a0-predictor-77df54fb5f from 0 to 1
18:25:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-b713a0	UpdateFailed	Failed to update status for InferenceService "isvc-init-fail-b713a0": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-b713a0": the object has been modified; please apply your changes to the latest version and try again
18:25:28	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-b713a0	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-b713a0": the object has been modified; please apply your changes to the latest version and try again
18:25:29	kserve-ci-e2e-test	multus	isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb	AddedInterface	Add eth0 [10.134.0.36/23] from ovn-kubernetes
18:25:33 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-init-fail-b713a0	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:25:33 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb	Started	Started container storage-initializer
18:25:33 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb	Created	Created container: storage-initializer
18:25:33 (x2)	kserve-ci-e2e-test	kubelet	isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d9a53871119e5163dc10506afed26a0cdd4198fd2315ea323b1e9fe09949b475" already present on machine
18:25:34	kserve-ci-e2e-test	kubelet	isvc-init-fail-b713a0-predictor-77df54fb5f-5nrvb	Killing	Stopping container storage-initializer
18:25:38	kserve-ci-e2e-test	deployment-controller	raw-sklearn-20421-predictor	ScalingReplicaSet	Scaled up replica set raw-sklearn-20421-predictor-58f54f4cf9 from 0 to 1
18:25:38	kserve-ci-e2e-test	v1beta1Controllers	raw-sklearn-20421	UpdateFailed	Failed to update status for InferenceService "raw-sklearn-20421": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-20421": the object has been modified; please apply your changes to the latest version and try again
18:25:39	kserve-ci-e2e-test	kubelet	raw-sklearn-20421-predictor-58f54f4cf9-czsvq	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d9a53871119e5163dc10506afed26a0cdd4198fd2315ea323b1e9fe09949b475" already present on machine
18:25:39	kserve-ci-e2e-test	multus	raw-sklearn-20421-predictor-58f54f4cf9-czsvq	AddedInterface	Add eth0 [10.134.0.37/23] from ovn-kubernetes
18:25:39	kserve-ci-e2e-test	kubelet	raw-sklearn-20421-predictor-58f54f4cf9-czsvq	Created	Created container: storage-initializer
18:25:39	kserve-ci-e2e-test	replicaset-controller	raw-sklearn-20421-predictor-58f54f4cf9	SuccessfulCreate	Created pod: raw-sklearn-20421-predictor-58f54f4cf9-czsvq
18:25:39	kserve-ci-e2e-test	kubelet	raw-sklearn-20421-predictor-58f54f4cf9-czsvq	Started	Started container storage-initializer
18:25:43	kserve-ci-e2e-test	kubelet	raw-sklearn-20421-predictor-58f54f4cf9-czsvq	Created	Created container: kserve-container
18:25:43	kserve-ci-e2e-test	kubelet	raw-sklearn-20421-predictor-58f54f4cf9-czsvq	Started	Started container kserve-container
18:25:43	kserve-ci-e2e-test	kubelet	raw-sklearn-20421-predictor-58f54f4cf9-czsvq	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:25:43	kserve-ci-e2e-test	kubelet	raw-sklearn-20421-predictor-58f54f4cf9-czsvq	Created	Created container: kube-rbac-proxy
18:25:43	kserve-ci-e2e-test	kubelet	raw-sklearn-20421-predictor-58f54f4cf9-czsvq	Started	Started container kube-rbac-proxy
18:25:43	kserve-ci-e2e-test	kubelet	raw-sklearn-20421-predictor-58f54f4cf9-czsvq	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1684" already present on machine
18:26:24 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	raw-sklearn-20421-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:26:24 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	raw-sklearn-20421-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:27:00	kserve-ci-e2e-test	v1beta1Controllers	raw-sklearn-20421	InferenceServiceReady	InferenceService [raw-sklearn-20421] is Ready
18:27:00 (x11)	kserve-ci-e2e-test	v1beta1Controllers	raw-sklearn-20421	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:27:09 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	raw-sklearn-20421-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:27:09 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	raw-sklearn-20421-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:27:09	kserve-ci-e2e-test	kubelet	raw-sklearn-20421-predictor-58f54f4cf9-czsvq	Killing	Stopping container kube-rbac-proxy
18:27:09	kserve-ci-e2e-test	kubelet	raw-sklearn-20421-predictor-58f54f4cf9-czsvq	Killing	Stopping container kserve-container
18:27:10	kserve-ci-e2e-test	kubelet	raw-sklearn-20421-predictor-58f54f4cf9-czsvq	Unhealthy	Readiness probe failed: Get "https://10.134.0.37:8643/healthz": dial tcp 10.134.0.37:8643: connect: connection refused
18:27:10 (x10)	kserve-ci-e2e-test	kubelet	raw-sklearn-20421-predictor-58f54f4cf9-czsvq	Unhealthy	Readiness probe failed: dial tcp 10.134.0.37:8080: connect: connection refused
18:27:14	kserve-ci-e2e-test	replicaset-controller	raw-sklearn-runtime-10c8f-predictor-649b8c799f	SuccessfulCreate	Created pod: raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6
18:27:14	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "raw-sklearn-runtime-10c8f-predictor-serving-cert" not found
18:27:14	kserve-ci-e2e-test	v1beta1Controllers	raw-sklearn-runtime-10c8f	UpdateFailed	Failed to update status for InferenceService "raw-sklearn-runtime-10c8f": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-10c8f": the object has been modified; please apply your changes to the latest version and try again
18:27:14	kserve-ci-e2e-test	deployment-controller	raw-sklearn-runtime-10c8f-predictor	ScalingReplicaSet	Scaled up replica set raw-sklearn-runtime-10c8f-predictor-649b8c799f from 0 to 1
18:27:15	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:d9a53871119e5163dc10506afed26a0cdd4198fd2315ea323b1e9fe09949b475" already present on machine
18:27:15	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6	Started	Started container storage-initializer
18:27:15	kserve-ci-e2e-test	multus	raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
18:27:15	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6	Created	Created container: storage-initializer
18:27:19	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6	Created	Created container: kserve-container
18:27:19	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6	Started	Started container kserve-container
18:27:19	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
18:27:19	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6	Created	Created container: kube-rbac-proxy
18:27:19	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6	Started	Started container kube-rbac-proxy
18:27:19	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6	Pulled	Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1684" already present on machine
18:27:44 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	raw-sklearn-runtime-10c8f-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:27:44 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	raw-sklearn-runtime-10c8f-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
18:28:31 (x10)	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6	Unhealthy	Readiness probe failed: dial tcp 10.134.0.38:8080: connect: connection refused
18:28:41	kserve-ci-e2e-test	v1beta1Controllers	raw-sklearn-runtime-10c8f	InferenceServiceReady	InferenceService [raw-sklearn-runtime-10c8f] is Ready
18:28:42 (x11)	kserve-ci-e2e-test	v1beta1Controllers	raw-sklearn-runtime-10c8f	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
18:28:44	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6	Killing	Stopping container kube-rbac-proxy
18:28:44	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6	Killing	Stopping container kserve-container
18:28:44 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	raw-sklearn-runtime-10c8f-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:28:44 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	raw-sklearn-runtime-10c8f-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:28:46	kserve-ci-e2e-test	kubelet	raw-sklearn-runtime-10c8f-predictor-649b8c799f-2hld6	Unhealthy	Readiness probe failed: Get "https://10.134.0.38:8643/healthz": dial tcp 10.134.0.38:8643: connect: connection refused