Events

Time	Namespace	Component	RelatedObject	Reason	Message
17:32:39	kserve-ci-e2e-test		isvc-xgboost-graph-predictor-555fc69d56-tbg9w	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-tbg9w to ip-10-0-142-249.ec2.internal
17:35:13	kserve-ci-e2e-test		switch-graph-22e0e-5d85974d85-2j87g	Scheduled	Successfully assigned kserve-ci-e2e-test/switch-graph-22e0e-5d85974d85-2j87g to ip-10-0-131-233.ec2.internal
17:35:23	kserve-ci-e2e-test		error-404-isvc-b1dfe-predictor-54cdc9876b-kc6bw	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-b1dfe-predictor-54cdc9876b-kc6bw to ip-10-0-141-165.ec2.internal
17:32:39	kserve-ci-e2e-test		isvc-sklearn-graph-1-predictor-84c58bd689-6fbfg	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-84c58bd689-6fbfg to ip-10-0-142-249.ec2.internal
17:45:38	kserve-ci-e2e-test		ensemble-graph-614a5-7dcdd75c-qs5jf	Scheduled	Successfully assigned kserve-ci-e2e-test/ensemble-graph-614a5-7dcdd75c-qs5jf to ip-10-0-131-233.ec2.internal
17:55:07	kserve-ci-e2e-test		success-200-isvc-cb1ee-predictor-7b757b875f-46rjc	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-cb1ee-predictor-7b757b875f-46rjc to ip-10-0-131-233.ec2.internal
17:32:39	kserve-ci-e2e-test		success-200-isvc-f0903-predictor-b88cb88c-8g2fn	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-f0903-predictor-b88cb88c-8g2fn to ip-10-0-131-233.ec2.internal
17:54:53	kserve-ci-e2e-test		splitter-graph-f7704-758c568bf-l924l	Scheduled	Successfully assigned kserve-ci-e2e-test/splitter-graph-f7704-758c568bf-l924l to ip-10-0-131-233.ec2.internal
17:55:04	kserve-ci-e2e-test		error-404-isvc-7d999-predictor-566585ff9d-rtwzr	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-7d999-predictor-566585ff9d-rtwzr to ip-10-0-141-165.ec2.internal
17:56:04	kserve-ci-e2e-test		splitter-graph-7d999-849c5688bd-rrgg7	Scheduled	Successfully assigned kserve-ci-e2e-test/splitter-graph-7d999-849c5688bd-rrgg7 to ip-10-0-131-233.ec2.internal
17:36:23	kserve-ci-e2e-test		sequence-graph-b1dfe-bf8f99978-2vj8v	Scheduled	Successfully assigned kserve-ci-e2e-test/sequence-graph-b1dfe-bf8f99978-2vj8v to ip-10-0-131-233.ec2.internal
17:46:52	kserve-ci-e2e-test		sequence-graph-787a1-77c7b956dc-cpngn	Scheduled	Successfully assigned kserve-ci-e2e-test/sequence-graph-787a1-77c7b956dc-cpngn to ip-10-0-131-233.ec2.internal
17:33:59	kserve-ci-e2e-test		switch-graph-f0903-56678cb99b-4nwlp	Scheduled	Successfully assigned kserve-ci-e2e-test/switch-graph-f0903-56678cb99b-4nwlp to ip-10-0-131-233.ec2.internal
17:56:17	kserve-ci-e2e-test		switch-graph-cb1ee-7ddbb59689-cbvqc	Scheduled	Successfully assigned kserve-ci-e2e-test/switch-graph-cb1ee-7ddbb59689-cbvqc to ip-10-0-131-233.ec2.internal
17:45:38	kserve-ci-e2e-test		sequence-graph-61a23-7dfbd4b7cf-ll569	Scheduled	Successfully assigned kserve-ci-e2e-test/sequence-graph-61a23-7dfbd4b7cf-ll569 to ip-10-0-131-233.ec2.internal
17:35:09	kserve-ci-e2e-test		model-chainer-64698579b9-v2bps	Scheduled	Successfully assigned kserve-ci-e2e-test/model-chainer-64698579b9-v2bps to ip-10-0-131-233.ec2.internal
17:32:39	kserve-ci-e2e-test		error-404-isvc-f0903-predictor-5f4f5b79bb-nlqzd	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-f0903-predictor-5f4f5b79bb-nlqzd to ip-10-0-141-165.ec2.internal
17:44:28	kserve-ci-e2e-test		ensemble-graph-8b8b5-846dd8bd57-vd26j	Scheduled	Successfully assigned kserve-ci-e2e-test/ensemble-graph-8b8b5-846dd8bd57-vd26j to ip-10-0-131-233.ec2.internal
17:44:38	kserve-ci-e2e-test		error-404-isvc-61a23-predictor-fd5b4c6d4-52d4w	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-61a23-predictor-fd5b4c6d4-52d4w to ip-10-0-141-165.ec2.internal
17:55:07	kserve-ci-e2e-test		error-404-isvc-cb1ee-predictor-8494ff7d66-sv48x	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-cb1ee-predictor-8494ff7d66-sv48x to ip-10-0-141-165.ec2.internal
17:53:53	kserve-ci-e2e-test		error-404-isvc-f7704-predictor-5b75c47dd6-jxdm4	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-f7704-predictor-5b75c47dd6-jxdm4 to ip-10-0-141-165.ec2.internal
17:43:28	kserve-ci-e2e-test		error-404-isvc-8b8b5-predictor-68f9fbd6b8-5djw5	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-8b8b5-predictor-68f9fbd6b8-5djw5 to ip-10-0-141-165.ec2.internal
17:35:23	kserve-ci-e2e-test		success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj to ip-10-0-131-233.ec2.internal
17:43:28	kserve-ci-e2e-test		success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk to ip-10-0-131-233.ec2.internal
17:55:04	kserve-ci-e2e-test		success-200-isvc-7d999-predictor-7f48846ddc-d5stl	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-7d999-predictor-7f48846ddc-d5stl to ip-10-0-131-233.ec2.internal
17:44:39	kserve-ci-e2e-test		error-404-isvc-614a5-predictor-55c87cb68d-g9s48	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-614a5-predictor-55c87cb68d-g9s48 to ip-10-0-141-165.ec2.internal
17:34:13	kserve-ci-e2e-test		success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4 to ip-10-0-131-233.ec2.internal
17:34:13	kserve-ci-e2e-test		error-404-isvc-22e0e-predictor-6bc85ccf58-kfk2k	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-22e0e-predictor-6bc85ccf58-kfk2k to ip-10-0-141-165.ec2.internal
17:44:39	kserve-ci-e2e-test		success-200-isvc-614a5-predictor-776c647d45-4xwq8	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-614a5-predictor-776c647d45-4xwq8 to ip-10-0-131-233.ec2.internal
17:45:52	kserve-ci-e2e-test		success-200-isvc-787a1-predictor-869dbdb759-qnbjk	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-787a1-predictor-869dbdb759-qnbjk to ip-10-0-131-233.ec2.internal
17:45:52	kserve-ci-e2e-test		error-404-isvc-787a1-predictor-fcb464fdb-zzl47	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-787a1-predictor-fcb464fdb-zzl47 to ip-10-0-141-165.ec2.internal
17:44:38	kserve-ci-e2e-test		success-200-isvc-61a23-predictor-644f77f65c-7gb8k	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-61a23-predictor-644f77f65c-7gb8k to ip-10-0-131-233.ec2.internal
17:32:39	kserve-ci-e2e-test		isvc-sklearn-graph-2-predictor-5bc9545b47-6bbpl	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-6bbpl to ip-10-0-142-249.ec2.internal
17:53:53	kserve-ci-e2e-test		success-200-isvc-f7704-predictor-865dcf7dff-bjhft	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-f7704-predictor-865dcf7dff-bjhft to ip-10-0-131-233.ec2.internal
17:32:38	kserve-ci-e2e-test	deployment-controller	success-200-isvc-f0903-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-f0903-predictor-b88cb88c from 0 to 1
17:32:38	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-f0903	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-f0903": the object has been modified; please apply your changes to the latest version and try again
17:32:38	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-f0903	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-f0903": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-f0903": the object has been modified; please apply your changes to the latest version and try again
17:32:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-1	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-graph-1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-1": the object has been modified; please apply your changes to the latest version and try again
17:32:39	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-graph-1-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-graph-1-predictor-84c58bd689 from 0 to 1
17:32:39	kserve-ci-e2e-test	kubelet	success-200-isvc-f0903-predictor-b88cb88c-8g2fn	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "success-200-isvc-f0903-predictor-serving-cert" not found
17:32:39	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-f0903-predictor-5f4f5b79bb	SuccessfulCreate	Created pod: error-404-isvc-f0903-predictor-5f4f5b79bb-nlqzd
17:32:39	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-graph-2-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-graph-2-predictor-5bc9545b47 from 0 to 1
17:32:39	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-f0903-predictor-b88cb88c	SuccessfulCreate	Created pod: success-200-isvc-f0903-predictor-b88cb88c-8g2fn
17:32:39	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-graph-2-predictor-5bc9545b47	SuccessfulCreate	Created pod: isvc-sklearn-graph-2-predictor-5bc9545b47-6bbpl
17:32:39	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-1	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-1": the object has been modified; please apply your changes to the latest version and try again
17:32:39	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-graph-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-graph-predictor-555fc69d56 from 0 to 1
17:32:39	kserve-ci-e2e-test	kubelet	error-404-isvc-f0903-predictor-5f4f5b79bb-nlqzd	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "error-404-isvc-f0903-predictor-serving-cert" not found
17:32:39	kserve-ci-e2e-test	deployment-controller	error-404-isvc-f0903-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-f0903-predictor-5f4f5b79bb from 0 to 1
17:32:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-84c58bd689-6fbfg	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
17:32:39	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-graph-1-predictor-84c58bd689	SuccessfulCreate	Created pod: isvc-sklearn-graph-1-predictor-84c58bd689-6fbfg
17:32:39	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-graph-predictor-555fc69d56	SuccessfulCreate	Created pod: isvc-xgboost-graph-predictor-555fc69d56-tbg9w
17:32:40	kserve-ci-e2e-test	kubelet	error-404-isvc-f0903-predictor-5f4f5b79bb-nlqzd	Pulling	Pulling image "quay.io/opendatahub/error-404-isvc:odh-pr-1821"
17:32:40	kserve-ci-e2e-test	multus	error-404-isvc-f0903-predictor-5f4f5b79bb-nlqzd	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
17:32:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-84c58bd689-6fbfg	Started	Started container storage-initializer
17:32:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-5bc9545b47-6bbpl	Started	Started container storage-initializer
17:32:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-5bc9545b47-6bbpl	Created	Created container: storage-initializer
17:32:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-5bc9545b47-6bbpl	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine
17:32:40	kserve-ci-e2e-test	multus	isvc-sklearn-graph-1-predictor-84c58bd689-6fbfg	AddedInterface	Add eth0 [10.132.0.22/23] from ovn-kubernetes
17:32:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-84c58bd689-6fbfg	Created	Created container: storage-initializer
17:32:40 (x2)	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-tbg9w	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-graph-predictor-serving-cert" not found
17:32:40	kserve-ci-e2e-test	kubelet	success-200-isvc-f0903-predictor-b88cb88c-8g2fn	Pulling	Pulling image "quay.io/opendatahub/success-200-isvc:odh-pr-1821"
17:32:40	kserve-ci-e2e-test	multus	isvc-sklearn-graph-2-predictor-5bc9545b47-6bbpl	AddedInterface	Add eth0 [10.132.0.24/23] from ovn-kubernetes
17:32:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-84c58bd689-6fbfg	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine
17:32:40	kserve-ci-e2e-test	multus	success-200-isvc-f0903-predictor-b88cb88c-8g2fn	AddedInterface	Add eth0 [10.133.0.14/23] from ovn-kubernetes
17:32:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-tbg9w	Created	Created container: storage-initializer
17:32:41	kserve-ci-e2e-test	multus	isvc-xgboost-graph-predictor-555fc69d56-tbg9w	AddedInterface	Add eth0 [10.132.0.23/23] from ovn-kubernetes
17:32:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-tbg9w	Started	Started container storage-initializer
17:32:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-tbg9w	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine
17:32:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-5bc9545b47-6bbpl	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821"
17:32:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-84c58bd689-6fbfg	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821"
17:32:46	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-tbg9w	Pulling	Pulling image "kserve/xgbserver:latest"
17:32:51	kserve-ci-e2e-test	kubelet	success-200-isvc-f0903-predictor-b88cb88c-8g2fn	Pulled	Successfully pulled image "quay.io/opendatahub/success-200-isvc:odh-pr-1821" in 11.596s (11.596s including waiting). Image size: 1334926941 bytes.
17:32:51	kserve-ci-e2e-test	kubelet	success-200-isvc-f0903-predictor-b88cb88c-8g2fn	Created	Created container: kserve-container
17:32:51	kserve-ci-e2e-test	kubelet	success-200-isvc-f0903-predictor-b88cb88c-8g2fn	Started	Started container kserve-container
17:32:51	kserve-ci-e2e-test	kubelet	success-200-isvc-f0903-predictor-b88cb88c-8g2fn	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
17:32:51	kserve-ci-e2e-test	kubelet	error-404-isvc-f0903-predictor-5f4f5b79bb-nlqzd	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
17:32:51	kserve-ci-e2e-test	kubelet	error-404-isvc-f0903-predictor-5f4f5b79bb-nlqzd	Started	Started container kserve-container
17:32:51	kserve-ci-e2e-test	kubelet	error-404-isvc-f0903-predictor-5f4f5b79bb-nlqzd	Created	Created container: kserve-container
17:32:51	kserve-ci-e2e-test	kubelet	error-404-isvc-f0903-predictor-5f4f5b79bb-nlqzd	Pulled	Successfully pulled image "quay.io/opendatahub/error-404-isvc:odh-pr-1821" in 11.242s (11.242s including waiting). Image size: 1333940313 bytes.
17:32:53	kserve-ci-e2e-test	kubelet	success-200-isvc-f0903-predictor-b88cb88c-8g2fn	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.233s (2.233s including waiting). Image size: 211946088 bytes.
17:32:54	kserve-ci-e2e-test	kubelet	error-404-isvc-f0903-predictor-5f4f5b79bb-nlqzd	Created	Created container: kube-rbac-proxy
17:32:54	kserve-ci-e2e-test	kubelet	success-200-isvc-f0903-predictor-b88cb88c-8g2fn	Created	Created container: kube-rbac-proxy
17:32:54	kserve-ci-e2e-test	kubelet	success-200-isvc-f0903-predictor-b88cb88c-8g2fn	Started	Started container kube-rbac-proxy
17:32:54	kserve-ci-e2e-test	kubelet	error-404-isvc-f0903-predictor-5f4f5b79bb-nlqzd	Started	Started container kube-rbac-proxy
17:32:54	kserve-ci-e2e-test	kubelet	error-404-isvc-f0903-predictor-5f4f5b79bb-nlqzd	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.167s (2.167s including waiting). Image size: 211946088 bytes.
17:33:09 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-f0903-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:33:09 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-f0903-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:33:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-5bc9545b47-6bbpl	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
17:33:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-84c58bd689-6fbfg	Created	Created container: kserve-container
17:33:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-5bc9545b47-6bbpl	Started	Started container kserve-container
17:33:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-5bc9545b47-6bbpl	Created	Created container: kserve-container
17:33:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-5bc9545b47-6bbpl	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821" in 28.204s (28.204s including waiting). Image size: 1555722147 bytes.
17:33:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-84c58bd689-6fbfg	Started	Started container kserve-container
17:33:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-84c58bd689-6fbfg	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
17:33:12	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-84c58bd689-6fbfg	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821" in 28.204s (28.204s including waiting). Image size: 1555722147 bytes.
17:33:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-tbg9w	Started	Started container kserve-container
17:33:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-tbg9w	Created	Created container: kserve-container
17:33:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-tbg9w	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
17:33:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-tbg9w	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 26.78s (26.78s including waiting). Image size: 1307355896 bytes.
17:33:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-5bc9545b47-6bbpl	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 3.005s (3.005s including waiting). Image size: 211946088 bytes.
17:33:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-tbg9w	Started	Started container kube-rbac-proxy
17:33:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-84c58bd689-6fbfg	Started	Started container kube-rbac-proxy
17:33:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-84c58bd689-6fbfg	Created	Created container: kube-rbac-proxy
17:33:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-84c58bd689-6fbfg	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 3.008s (3.008s including waiting). Image size: 211946088 bytes.
17:33:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-tbg9w	Created	Created container: kube-rbac-proxy
17:33:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-tbg9w	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.435s (2.435s including waiting). Image size: 211946088 bytes.
17:33:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-5bc9545b47-6bbpl	Created	Created container: kube-rbac-proxy
17:33:15	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-5bc9545b47-6bbpl	Started	Started container kube-rbac-proxy
17:33:24 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:33:24 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:33:30 (x6)	kserve-ci-e2e-test	kubelet	success-200-isvc-f0903-predictor-b88cb88c-8g2fn	Unhealthy	Readiness probe failed: dial tcp 10.133.0.14:8080: connect: connection refused
17:33:31 (x6)	kserve-ci-e2e-test	kubelet	error-404-isvc-f0903-predictor-5f4f5b79bb-nlqzd	Unhealthy	Readiness probe failed: dial tcp 10.134.0.38:8080: connect: connection refused
17:33:39 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-f0903-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:33:39 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-f0903-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:33:40	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-f0903	InferenceServiceReady	InferenceService [success-200-isvc-f0903] is Ready
17:33:40 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-f0903	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:33:41 (x5)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-f0903	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:33:41	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-f0903	InferenceServiceReady	InferenceService [error-404-isvc-f0903] is Ready
17:33:54 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:33:54 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:33:54 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:33:54 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:33:59	kserve-ci-e2e-test	replicaset-controller	switch-graph-f0903-56678cb99b	SuccessfulCreate	Created pod: switch-graph-f0903-56678cb99b-4nwlp
17:33:59	kserve-ci-e2e-test	deployment-controller	switch-graph-f0903	ScalingReplicaSet	Scaled up replica set switch-graph-f0903-56678cb99b from 0 to 1
17:33:59	kserve-ci-e2e-test	InferenceGraphController	switch-graph-f0903	InferenceGraphReady	InferenceGraph [switch-graph-f0903] is Ready
17:33:59	kserve-ci-e2e-test	kubelet	switch-graph-f0903-56678cb99b-4nwlp	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "switch-graph-f0903-serving-cert" not found
17:34:00	kserve-ci-e2e-test	kubelet	switch-graph-f0903-56678cb99b-4nwlp	Pulling	Pulling image "quay.io/opendatahub/kserve-router@sha256:2d78b1e75ce4a3e633713c1f964bde1c2425164638b26ad77af20d6340c844d7"
17:34:00	kserve-ci-e2e-test	multus	switch-graph-f0903-56678cb99b-4nwlp	AddedInterface	Add eth0 [10.133.0.15/23] from ovn-kubernetes
17:34:02	kserve-ci-e2e-test	kubelet	switch-graph-f0903-56678cb99b-4nwlp	Created	Created container: switch-graph-f0903
17:34:02	kserve-ci-e2e-test	kubelet	switch-graph-f0903-56678cb99b-4nwlp	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:2d78b1e75ce4a3e633713c1f964bde1c2425164638b26ad77af20d6340c844d7" in 2.343s (2.343s including waiting). Image size: 218226779 bytes.
17:34:02	kserve-ci-e2e-test	kubelet	switch-graph-f0903-56678cb99b-4nwlp	Started	Started container switch-graph-f0903
17:34:09 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-f0903-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:34:09 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-f0903-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:34:09 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-f0903-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:34:09 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-f0903-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:34:12 (x8)	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-tbg9w	Unhealthy	Readiness probe failed: dial tcp 10.132.0.23:8080: connect: connection refused
17:34:13	kserve-ci-e2e-test	kubelet	error-404-isvc-f0903-predictor-5f4f5b79bb-nlqzd	Killing	Stopping container kube-rbac-proxy
17:34:13	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-22e0e	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-22e0e": the object has been modified; please apply your changes to the latest version and try again
17:34:13	kserve-ci-e2e-test	kubelet	success-200-isvc-f0903-predictor-b88cb88c-8g2fn	Killing	Stopping container kserve-container
17:34:13	kserve-ci-e2e-test	kubelet	error-404-isvc-22e0e-predictor-6bc85ccf58-kfk2k	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "error-404-isvc-22e0e-predictor-serving-cert" not found
17:34:13	kserve-ci-e2e-test	kubelet	success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1821" already present on machine
17:34:13	kserve-ci-e2e-test	kubelet	success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4	Created	Created container: kserve-container
17:34:13	kserve-ci-e2e-test	deployment-controller	success-200-isvc-22e0e-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-22e0e-predictor-dcf6f766c from 0 to 1
17:34:13	kserve-ci-e2e-test	kubelet	error-404-isvc-f0903-predictor-5f4f5b79bb-nlqzd	Killing	Stopping container kserve-container
17:34:13	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-22e0e	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-22e0e": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-22e0e": the object has been modified; please apply your changes to the latest version and try again
17:34:13	kserve-ci-e2e-test	kubelet	switch-graph-f0903-56678cb99b-4nwlp	Killing	Stopping container switch-graph-f0903
17:34:13	kserve-ci-e2e-test	kubelet	success-200-isvc-f0903-predictor-b88cb88c-8g2fn	Killing	Stopping container kube-rbac-proxy
17:34:13	kserve-ci-e2e-test	multus	success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4	AddedInterface	Add eth0 [10.133.0.16/23] from ovn-kubernetes
17:34:13	kserve-ci-e2e-test	kubelet	success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4	Started	Started container kserve-container
17:34:13	kserve-ci-e2e-test	kubelet	success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:34:13	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-22e0e-predictor-dcf6f766c	SuccessfulCreate	Created pod: success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4
17:34:13	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-22e0e-predictor-6bc85ccf58	SuccessfulCreate	Created pod: error-404-isvc-22e0e-predictor-6bc85ccf58-kfk2k
17:34:13	kserve-ci-e2e-test	deployment-controller	error-404-isvc-22e0e-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-22e0e-predictor-6bc85ccf58 from 0 to 1
17:34:13	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-22e0e	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-22e0e": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-22e0e": the object has been modified; please apply your changes to the latest version and try again
17:34:13	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-22e0e	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-22e0e": the object has been modified; please apply your changes to the latest version and try again
17:34:14	kserve-ci-e2e-test	kubelet	error-404-isvc-22e0e-predictor-6bc85ccf58-kfk2k	Created	Created container: kserve-container
17:34:14	kserve-ci-e2e-test	kubelet	success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4	Started	Started container kube-rbac-proxy
17:34:14	kserve-ci-e2e-test	kubelet	success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4	Created	Created container: kube-rbac-proxy
17:34:14	kserve-ci-e2e-test	kubelet	error-404-isvc-22e0e-predictor-6bc85ccf58-kfk2k	Started	Started container kube-rbac-proxy
17:34:14	kserve-ci-e2e-test	kubelet	error-404-isvc-22e0e-predictor-6bc85ccf58-kfk2k	Created	Created container: kube-rbac-proxy
17:34:14	kserve-ci-e2e-test	kubelet	error-404-isvc-22e0e-predictor-6bc85ccf58-kfk2k	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:34:14	kserve-ci-e2e-test	kubelet	error-404-isvc-22e0e-predictor-6bc85ccf58-kfk2k	Started	Started container kserve-container
17:34:14	kserve-ci-e2e-test	kubelet	error-404-isvc-22e0e-predictor-6bc85ccf58-kfk2k	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1821" already present on machine
17:34:14	kserve-ci-e2e-test	multus	error-404-isvc-22e0e-predictor-6bc85ccf58-kfk2k	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
17:34:15	kserve-ci-e2e-test	kubelet	success-200-isvc-f0903-predictor-b88cb88c-8g2fn	Unhealthy	Readiness probe failed: Get "https://10.133.0.14:8643/healthz": dial tcp 10.133.0.14:8643: connect: connection refused
17:34:16	kserve-ci-e2e-test	kubelet	error-404-isvc-f0903-predictor-5f4f5b79bb-nlqzd	Unhealthy	Readiness probe failed: Get "https://10.134.0.38:8643/healthz": dial tcp 10.134.0.38:8643: connect: connection refused
17:34:22 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-graph	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:34:22	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-graph	InferenceServiceReady	InferenceService [isvc-xgboost-graph] is Ready
17:34:22 (x9)	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-5bc9545b47-6bbpl	Unhealthy	Readiness probe failed: dial tcp 10.132.0.24:8080: connect: connection refused
17:34:28 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-84c58bd689-6fbfg	Unhealthy	Readiness probe failed: dial tcp 10.132.0.22:8080: connect: connection refused
17:34:31	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-2	InferenceServiceReady	InferenceService [isvc-sklearn-graph-2] is Ready
17:34:32 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:34:38	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-1	InferenceServiceReady	InferenceService [isvc-sklearn-graph-1] is Ready
17:34:39 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:34:39 (x6)	kserve-ci-e2e-test	kubelet	switch-graph-f0903-56678cb99b-4nwlp	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
17:34:54 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:34:54 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:34:54 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:34:54 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:34:58 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-22e0e-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:34:58 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-22e0e-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:34:58 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-22e0e-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:34:58 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-22e0e-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:35:00	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-22e0e	InferenceServiceReady	InferenceService [error-404-isvc-22e0e] is Ready
17:35:00 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-22e0e	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:35:00	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-22e0e	InferenceServiceReady	InferenceService [success-200-isvc-22e0e] is Ready
17:35:01 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-22e0e	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:35:08	kserve-ci-e2e-test	deployment-controller	model-chainer	ScalingReplicaSet	Scaled up replica set model-chainer-64698579b9 from 0 to 1
17:35:09	kserve-ci-e2e-test	kubelet	model-chainer-64698579b9-v2bps	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-serving-cert" not found
17:35:09	kserve-ci-e2e-test	InferenceGraphController	model-chainer	UpdateFailed	Failed to update status for InferenceGraph "model-chainer": Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "model-chainer": the object has been modified; please apply your changes to the latest version and try again
17:35:09	kserve-ci-e2e-test	InferenceGraphController	model-chainer	InternalError	fails to update InferenceGraph status: Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "model-chainer": the object has been modified; please apply your changes to the latest version and try again
17:35:09 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:35:09 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:35:09	kserve-ci-e2e-test	InferenceGraphController	model-chainer	InferenceGraphReady	InferenceGraph [model-chainer] is Ready
17:35:09	kserve-ci-e2e-test	replicaset-controller	model-chainer-64698579b9	SuccessfulCreate	Created pod: model-chainer-64698579b9-v2bps
17:35:10	kserve-ci-e2e-test	kubelet	model-chainer-64698579b9-v2bps	Started	Started container model-chainer
17:35:10	kserve-ci-e2e-test	kubelet	model-chainer-64698579b9-v2bps	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:2d78b1e75ce4a3e633713c1f964bde1c2425164638b26ad77af20d6340c844d7" already present on machine
17:35:10	kserve-ci-e2e-test	multus	model-chainer-64698579b9-v2bps	AddedInterface	Add eth0 [10.133.0.17/23] from ovn-kubernetes
17:35:10	kserve-ci-e2e-test	kubelet	model-chainer-64698579b9-v2bps	Created	Created container: model-chainer
17:35:13	kserve-ci-e2e-test	deployment-controller	switch-graph-22e0e	ScalingReplicaSet	Scaled up replica set switch-graph-22e0e-5d85974d85 from 0 to 1
17:35:13	kserve-ci-e2e-test	InferenceGraphController	switch-graph-22e0e	InternalError	fails to update InferenceGraph status: Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "switch-graph-22e0e": the object has been modified; please apply your changes to the latest version and try again
17:35:13	kserve-ci-e2e-test	InferenceGraphController	switch-graph-22e0e	InferenceGraphReady	InferenceGraph [switch-graph-22e0e] is Ready
17:35:13	kserve-ci-e2e-test	InferenceGraphController	switch-graph-22e0e	UpdateFailed	Failed to update status for InferenceGraph "switch-graph-22e0e": Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "switch-graph-22e0e": the object has been modified; please apply your changes to the latest version and try again
17:35:13	kserve-ci-e2e-test	replicaset-controller	switch-graph-22e0e-5d85974d85	SuccessfulCreate	Created pod: switch-graph-22e0e-5d85974d85-2j87g
17:35:13	kserve-ci-e2e-test	kubelet	switch-graph-22e0e-5d85974d85-2j87g	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "switch-graph-22e0e-serving-cert" not found
17:35:14	kserve-ci-e2e-test	kubelet	switch-graph-22e0e-5d85974d85-2j87g	Started	Started container switch-graph-22e0e
17:35:14	kserve-ci-e2e-test	kubelet	switch-graph-22e0e-5d85974d85-2j87g	Created	Created container: switch-graph-22e0e
17:35:14	kserve-ci-e2e-test	kubelet	switch-graph-22e0e-5d85974d85-2j87g	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:2d78b1e75ce4a3e633713c1f964bde1c2425164638b26ad77af20d6340c844d7" already present on machine
17:35:14	kserve-ci-e2e-test	multus	switch-graph-22e0e-5d85974d85-2j87g	AddedInterface	Add eth0 [10.133.0.18/23] from ovn-kubernetes
17:35:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-tbg9w	Killing	Stopping container kserve-container
17:35:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-5bc9545b47-6bbpl	Killing	Stopping container kserve-container
17:35:23	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-b1dfe-predictor-54cdc9876b	SuccessfulCreate	Created pod: error-404-isvc-b1dfe-predictor-54cdc9876b-kc6bw
17:35:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-tbg9w	Killing	Stopping container kube-rbac-proxy
17:35:23	kserve-ci-e2e-test	deployment-controller	error-404-isvc-b1dfe-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-b1dfe-predictor-54cdc9876b from 0 to 1
17:35:23	kserve-ci-e2e-test	multus	success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj	AddedInterface	Add eth0 [10.133.0.19/23] from ovn-kubernetes
17:35:23	kserve-ci-e2e-test	kubelet	success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1821" already present on machine
17:35:23	kserve-ci-e2e-test	kubelet	success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj	Created	Created container: kserve-container
17:35:23	kserve-ci-e2e-test	kubelet	success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj	Started	Started container kserve-container
17:35:23	kserve-ci-e2e-test	kubelet	success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:35:23	kserve-ci-e2e-test	kubelet	success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj	Created	Created container: kube-rbac-proxy
17:35:23	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-b1dfe	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-b1dfe": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-b1dfe": the object has been modified; please apply your changes to the latest version and try again
17:35:23	kserve-ci-e2e-test	kubelet	model-chainer-64698579b9-v2bps	Killing	Stopping container model-chainer
17:35:23	kserve-ci-e2e-test	kubelet	success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj	Started	Started container kube-rbac-proxy
17:35:23	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-b1dfe-predictor-559469fcc9	SuccessfulCreate	Created pod: success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj
17:35:23	kserve-ci-e2e-test	deployment-controller	success-200-isvc-b1dfe-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-b1dfe-predictor-559469fcc9 from 0 to 1
17:35:23	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-b1dfe	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-b1dfe": the object has been modified; please apply your changes to the latest version and try again
17:35:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-84c58bd689-6fbfg	Killing	Stopping container kserve-container
17:35:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-84c58bd689-6fbfg	Killing	Stopping container kube-rbac-proxy
17:35:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-84c58bd689-6fbfg	Unhealthy	Readiness probe failed: Get "https://10.132.0.22:8643/healthz": dial tcp 10.132.0.22:8643: connect: connection refused
17:35:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-5bc9545b47-6bbpl	Killing	Stopping container kube-rbac-proxy
17:35:24 (x2)	kserve-ci-e2e-test	kubelet	error-404-isvc-b1dfe-predictor-54cdc9876b-kc6bw	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "error-404-isvc-b1dfe-predictor-serving-cert" not found
17:35:25	kserve-ci-e2e-test	kubelet	error-404-isvc-b1dfe-predictor-54cdc9876b-kc6bw	Started	Started container kserve-container
17:35:25	kserve-ci-e2e-test	kubelet	error-404-isvc-b1dfe-predictor-54cdc9876b-kc6bw	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:35:25	kserve-ci-e2e-test	kubelet	error-404-isvc-b1dfe-predictor-54cdc9876b-kc6bw	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1821" already present on machine
17:35:25	kserve-ci-e2e-test	kubelet	error-404-isvc-b1dfe-predictor-54cdc9876b-kc6bw	Started	Started container kube-rbac-proxy
17:35:25	kserve-ci-e2e-test	kubelet	error-404-isvc-b1dfe-predictor-54cdc9876b-kc6bw	Created	Created container: kserve-container
17:35:25	kserve-ci-e2e-test	multus	error-404-isvc-b1dfe-predictor-54cdc9876b-kc6bw	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
17:35:25	kserve-ci-e2e-test	kubelet	error-404-isvc-b1dfe-predictor-54cdc9876b-kc6bw	Created	Created container: kube-rbac-proxy
17:35:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-5bc9545b47-6bbpl	Unhealthy	Readiness probe failed: Get "https://10.132.0.24:8643/healthz": dial tcp 10.132.0.24:8643: connect: connection refused
17:35:28 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-22e0e-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:35:28 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-22e0e-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:35:38	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-b1dfe-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:35:38	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-b1dfe-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:35:43 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-22e0e-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:35:43 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-22e0e-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:35:51 (x6)	kserve-ci-e2e-test	kubelet	model-chainer-64698579b9-v2bps	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
17:35:58 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-22e0e	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:35:58 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-22e0e	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:36:03 (x6)	kserve-ci-e2e-test	kubelet	error-404-isvc-b1dfe-predictor-54cdc9876b-kc6bw	Unhealthy	Readiness probe failed: dial tcp 10.134.0.40:8080: connect: connection refused
17:36:08 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-b1dfe-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:36:08 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-b1dfe-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:36:10 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-b1dfe	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:36:10	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-b1dfe	InferenceServiceReady	InferenceService [success-200-isvc-b1dfe] is Ready
17:36:13	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-b1dfe	InferenceServiceReady	InferenceService [error-404-isvc-b1dfe] is Ready
17:36:13 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-b1dfe	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:36:23	kserve-ci-e2e-test	kubelet	sequence-graph-b1dfe-bf8f99978-2vj8v	Created	Created container: sequence-graph-b1dfe
17:36:23	kserve-ci-e2e-test	multus	sequence-graph-b1dfe-bf8f99978-2vj8v	AddedInterface	Add eth0 [10.133.0.20/23] from ovn-kubernetes
17:36:23	kserve-ci-e2e-test	replicaset-controller	sequence-graph-b1dfe-bf8f99978	SuccessfulCreate	Created pod: sequence-graph-b1dfe-bf8f99978-2vj8v
17:36:23	kserve-ci-e2e-test	deployment-controller	sequence-graph-b1dfe	ScalingReplicaSet	Scaled up replica set sequence-graph-b1dfe-bf8f99978 from 0 to 1
17:36:23	kserve-ci-e2e-test	kubelet	sequence-graph-b1dfe-bf8f99978-2vj8v	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:2d78b1e75ce4a3e633713c1f964bde1c2425164638b26ad77af20d6340c844d7" already present on machine
17:36:23	kserve-ci-e2e-test	InferenceGraphController	sequence-graph-b1dfe	InferenceGraphReady	InferenceGraph [sequence-graph-b1dfe] is Ready
17:36:23	kserve-ci-e2e-test	kubelet	sequence-graph-b1dfe-bf8f99978-2vj8v	Started	Started container sequence-graph-b1dfe
17:36:38	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-b1dfe	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:36:38 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-b1dfe-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:36:38 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-b1dfe-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:36:38	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-b1dfe	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:36:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-b1dfe-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:36:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-b1dfe-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:36:53	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-b1dfe	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:36:53	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-b1dfe	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:43:28	kserve-ci-e2e-test	deployment-controller	success-200-isvc-8b8b5-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-8b8b5-predictor-7f4bbc59f7 from 0 to 1
17:43:28	kserve-ci-e2e-test	kubelet	error-404-isvc-22e0e-predictor-6bc85ccf58-kfk2k	Killing	Stopping container kube-rbac-proxy
17:43:28	kserve-ci-e2e-test	kubelet	success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "success-200-isvc-8b8b5-predictor-serving-cert" not found
17:43:28	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-8b8b5-predictor-68f9fbd6b8	SuccessfulCreate	Created pod: error-404-isvc-8b8b5-predictor-68f9fbd6b8-5djw5
17:43:28	kserve-ci-e2e-test	deployment-controller	error-404-isvc-8b8b5-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-8b8b5-predictor-68f9fbd6b8 from 0 to 1
17:43:28	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-8b8b5	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-8b8b5": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-8b8b5": the object has been modified; please apply your changes to the latest version and try again
17:43:28	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-8b8b5	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-8b8b5": the object has been modified; please apply your changes to the latest version and try again
17:43:28	kserve-ci-e2e-test	kubelet	error-404-isvc-8b8b5-predictor-68f9fbd6b8-5djw5	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "error-404-isvc-8b8b5-predictor-serving-cert" not found
17:43:28	kserve-ci-e2e-test	kubelet	switch-graph-22e0e-5d85974d85-2j87g	Killing	Stopping container switch-graph-22e0e
17:43:28	kserve-ci-e2e-test	kubelet	success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4	Killing	Stopping container kube-rbac-proxy
17:43:28	kserve-ci-e2e-test	kubelet	success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4	Killing	Stopping container kserve-container
17:43:28	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-8b8b5-predictor-7f4bbc59f7	SuccessfulCreate	Created pod: success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk
17:43:28	kserve-ci-e2e-test	kubelet	error-404-isvc-22e0e-predictor-6bc85ccf58-kfk2k	Killing	Stopping container kserve-container
17:43:28	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-8b8b5	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-8b8b5": the object has been modified; please apply your changes to the latest version and try again
17:43:28	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-8b8b5	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-8b8b5": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-8b8b5": the object has been modified; please apply your changes to the latest version and try again
17:43:29	kserve-ci-e2e-test	kubelet	success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk	Started	Started container kserve-container
17:43:29	kserve-ci-e2e-test	kubelet	error-404-isvc-8b8b5-predictor-68f9fbd6b8-5djw5	Created	Created container: kube-rbac-proxy
17:43:29	kserve-ci-e2e-test	multus	success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk	AddedInterface	Add eth0 [10.133.0.21/23] from ovn-kubernetes
17:43:29	kserve-ci-e2e-test	kubelet	error-404-isvc-8b8b5-predictor-68f9fbd6b8-5djw5	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:43:29	kserve-ci-e2e-test	kubelet	error-404-isvc-8b8b5-predictor-68f9fbd6b8-5djw5	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1821" already present on machine
17:43:29	kserve-ci-e2e-test	multus	error-404-isvc-8b8b5-predictor-68f9fbd6b8-5djw5	AddedInterface	Add eth0 [10.134.0.41/23] from ovn-kubernetes
17:43:29	kserve-ci-e2e-test	kubelet	success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk	Created	Created container: kube-rbac-proxy
17:43:29	kserve-ci-e2e-test	kubelet	error-404-isvc-8b8b5-predictor-68f9fbd6b8-5djw5	Started	Started container kserve-container
17:43:29	kserve-ci-e2e-test	kubelet	error-404-isvc-8b8b5-predictor-68f9fbd6b8-5djw5	Created	Created container: kserve-container
17:43:29	kserve-ci-e2e-test	kubelet	success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk	Created	Created container: kserve-container
17:43:29	kserve-ci-e2e-test	kubelet	error-404-isvc-8b8b5-predictor-68f9fbd6b8-5djw5	Started	Started container kube-rbac-proxy
17:43:29	kserve-ci-e2e-test	kubelet	success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1821" already present on machine
17:43:29	kserve-ci-e2e-test	kubelet	success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk	Started	Started container kube-rbac-proxy
17:43:29	kserve-ci-e2e-test	kubelet	success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:43:30	kserve-ci-e2e-test	kubelet	error-404-isvc-22e0e-predictor-6bc85ccf58-kfk2k	Unhealthy	Readiness probe failed: Get "https://10.134.0.39:8643/healthz": dial tcp 10.134.0.39:8643: connect: connection refused
17:43:30	kserve-ci-e2e-test	kubelet	success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4	Unhealthy	Readiness probe failed: Get "https://10.133.0.16:8643/healthz": dial tcp 10.133.0.16:8643: connect: connection refused
17:43:30 (x7)	kserve-ci-e2e-test	kubelet	success-200-isvc-22e0e-predictor-dcf6f766c-xcbz4	Unhealthy	Readiness probe failed: dial tcp 10.133.0.16:8080: connect: connection refused
17:43:30 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-22e0e-predictor-6bc85ccf58-kfk2k	Unhealthy	Readiness probe failed: dial tcp 10.134.0.39:8080: connect: connection refused
17:43:55 (x6)	kserve-ci-e2e-test	kubelet	switch-graph-22e0e-5d85974d85-2j87g	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
17:44:07 (x6)	kserve-ci-e2e-test	kubelet	success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk	Unhealthy	Readiness probe failed: dial tcp 10.133.0.21:8080: connect: connection refused
17:44:07 (x6)	kserve-ci-e2e-test	kubelet	error-404-isvc-8b8b5-predictor-68f9fbd6b8-5djw5	Unhealthy	Readiness probe failed: dial tcp 10.134.0.41:8080: connect: connection refused
17:44:13 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-8b8b5-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:44:13 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-8b8b5-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:44:13 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-8b8b5-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:44:13 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-8b8b5-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:44:17 (x6)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-8b8b5	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:44:17 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-8b8b5	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:44:17	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-8b8b5	InferenceServiceReady	InferenceService [error-404-isvc-8b8b5] is Ready
17:44:17	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-8b8b5	InferenceServiceReady	InferenceService [success-200-isvc-8b8b5] is Ready
17:44:28 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-8b8b5-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:44:28	kserve-ci-e2e-test	InferenceGraphController	ensemble-graph-8b8b5	InferenceGraphReady	InferenceGraph [ensemble-graph-8b8b5] is Ready
17:44:28 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-8b8b5-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:44:28 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-8b8b5-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:44:28 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-8b8b5-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:44:28	kserve-ci-e2e-test	multus	ensemble-graph-8b8b5-846dd8bd57-vd26j	AddedInterface	Add eth0 [10.133.0.22/23] from ovn-kubernetes
17:44:28	kserve-ci-e2e-test	kubelet	ensemble-graph-8b8b5-846dd8bd57-vd26j	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:2d78b1e75ce4a3e633713c1f964bde1c2425164638b26ad77af20d6340c844d7" already present on machine
17:44:28	kserve-ci-e2e-test	kubelet	ensemble-graph-8b8b5-846dd8bd57-vd26j	Created	Created container: ensemble-graph-8b8b5
17:44:28	kserve-ci-e2e-test	kubelet	ensemble-graph-8b8b5-846dd8bd57-vd26j	Started	Started container ensemble-graph-8b8b5
17:44:28	kserve-ci-e2e-test	replicaset-controller	ensemble-graph-8b8b5-846dd8bd57	SuccessfulCreate	Created pod: ensemble-graph-8b8b5-846dd8bd57-vd26j
17:44:28	kserve-ci-e2e-test	deployment-controller	ensemble-graph-8b8b5	ScalingReplicaSet	Scaled up replica set ensemble-graph-8b8b5-846dd8bd57 from 0 to 1
17:44:37	kserve-ci-e2e-test	kubelet	sequence-graph-b1dfe-bf8f99978-2vj8v	Killing	Stopping container sequence-graph-b1dfe
17:44:38	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-61a23	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-61a23": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-61a23": the object has been modified; please apply your changes to the latest version and try again
17:44:38	kserve-ci-e2e-test	multus	error-404-isvc-61a23-predictor-fd5b4c6d4-52d4w	AddedInterface	Add eth0 [10.134.0.42/23] from ovn-kubernetes
17:44:38	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-61a23	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-61a23": the object has been modified; please apply your changes to the latest version and try again
17:44:38	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-61a23	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-61a23": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-61a23": the object has been modified; please apply your changes to the latest version and try again
17:44:38	kserve-ci-e2e-test	kubelet	ensemble-graph-8b8b5-846dd8bd57-vd26j	Killing	Stopping container ensemble-graph-8b8b5
17:44:38	kserve-ci-e2e-test	kubelet	error-404-isvc-b1dfe-predictor-54cdc9876b-kc6bw	Killing	Stopping container kube-rbac-proxy
17:44:38	kserve-ci-e2e-test	kubelet	error-404-isvc-b1dfe-predictor-54cdc9876b-kc6bw	Killing	Stopping container kserve-container
17:44:38	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-61a23	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-61a23": the object has been modified; please apply your changes to the latest version and try again
17:44:38	kserve-ci-e2e-test	kubelet	success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj	Killing	Stopping container kube-rbac-proxy
17:44:38	kserve-ci-e2e-test	kubelet	success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj	Killing	Stopping container kserve-container
17:44:38	kserve-ci-e2e-test	deployment-controller	success-200-isvc-61a23-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-61a23-predictor-644f77f65c from 0 to 1
17:44:38	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-61a23-predictor-644f77f65c	SuccessfulCreate	Created pod: success-200-isvc-61a23-predictor-644f77f65c-7gb8k
17:44:38	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-61a23-predictor-fd5b4c6d4	SuccessfulCreate	Created pod: error-404-isvc-61a23-predictor-fd5b4c6d4-52d4w
17:44:38	kserve-ci-e2e-test	deployment-controller	success-200-isvc-614a5-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-614a5-predictor-776c647d45 from 0 to 1
17:44:38 (x2)	kserve-ci-e2e-test	kubelet	success-200-isvc-61a23-predictor-644f77f65c-7gb8k	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "success-200-isvc-61a23-predictor-serving-cert" not found
17:44:38	kserve-ci-e2e-test	kubelet	error-404-isvc-61a23-predictor-fd5b4c6d4-52d4w	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1821" already present on machine
17:44:38	kserve-ci-e2e-test	deployment-controller	error-404-isvc-61a23-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-61a23-predictor-fd5b4c6d4 from 0 to 1
17:44:39	kserve-ci-e2e-test	kubelet	error-404-isvc-61a23-predictor-fd5b4c6d4-52d4w	Started	Started container kube-rbac-proxy
17:44:39	kserve-ci-e2e-test	multus	success-200-isvc-614a5-predictor-776c647d45-4xwq8	AddedInterface	Add eth0 [10.133.0.24/23] from ovn-kubernetes
17:44:39	kserve-ci-e2e-test	kubelet	success-200-isvc-614a5-predictor-776c647d45-4xwq8	Started	Started container kserve-container
17:44:39	kserve-ci-e2e-test	kubelet	error-404-isvc-614a5-predictor-55c87cb68d-g9s48	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1821" already present on machine
17:44:39	kserve-ci-e2e-test	kubelet	error-404-isvc-614a5-predictor-55c87cb68d-g9s48	Created	Created container: kserve-container
17:44:39	kserve-ci-e2e-test	kubelet	success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk	Killing	Stopping container kube-rbac-proxy
17:44:39	kserve-ci-e2e-test	kubelet	success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk	Killing	Stopping container kserve-container
17:44:39	kserve-ci-e2e-test	kubelet	error-404-isvc-614a5-predictor-55c87cb68d-g9s48	Started	Started container kserve-container
17:44:39	kserve-ci-e2e-test	kubelet	error-404-isvc-614a5-predictor-55c87cb68d-g9s48	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:44:39	kserve-ci-e2e-test	kubelet	error-404-isvc-614a5-predictor-55c87cb68d-g9s48	Created	Created container: kube-rbac-proxy
17:44:39	kserve-ci-e2e-test	kubelet	error-404-isvc-61a23-predictor-fd5b4c6d4-52d4w	Created	Created container: kserve-container
17:44:39	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-614a5-predictor-776c647d45	SuccessfulCreate	Created pod: success-200-isvc-614a5-predictor-776c647d45-4xwq8
17:44:39	kserve-ci-e2e-test	kubelet	error-404-isvc-614a5-predictor-55c87cb68d-g9s48	Started	Started container kube-rbac-proxy
17:44:39	kserve-ci-e2e-test	multus	error-404-isvc-614a5-predictor-55c87cb68d-g9s48	AddedInterface	Add eth0 [10.134.0.43/23] from ovn-kubernetes
17:44:39	kserve-ci-e2e-test	kubelet	success-200-isvc-614a5-predictor-776c647d45-4xwq8	Started	Started container kube-rbac-proxy
17:44:39	kserve-ci-e2e-test	kubelet	success-200-isvc-614a5-predictor-776c647d45-4xwq8	Created	Created container: kube-rbac-proxy
17:44:39	kserve-ci-e2e-test	kubelet	error-404-isvc-8b8b5-predictor-68f9fbd6b8-5djw5	Killing	Stopping container kserve-container
17:44:39	kserve-ci-e2e-test	deployment-controller	error-404-isvc-614a5-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-614a5-predictor-55c87cb68d from 0 to 1
17:44:39	kserve-ci-e2e-test	kubelet	success-200-isvc-614a5-predictor-776c647d45-4xwq8	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1821" already present on machine
17:44:39	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-614a5-predictor-55c87cb68d	SuccessfulCreate	Created pod: error-404-isvc-614a5-predictor-55c87cb68d-g9s48
17:44:39	kserve-ci-e2e-test	kubelet	error-404-isvc-8b8b5-predictor-68f9fbd6b8-5djw5	Killing	Stopping container kube-rbac-proxy
17:44:39	kserve-ci-e2e-test	kubelet	error-404-isvc-61a23-predictor-fd5b4c6d4-52d4w	Started	Started container kserve-container
17:44:39	kserve-ci-e2e-test	kubelet	success-200-isvc-614a5-predictor-776c647d45-4xwq8	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:44:39	kserve-ci-e2e-test	kubelet	success-200-isvc-614a5-predictor-776c647d45-4xwq8	Created	Created container: kserve-container
17:44:39	kserve-ci-e2e-test	kubelet	error-404-isvc-61a23-predictor-fd5b4c6d4-52d4w	Created	Created container: kube-rbac-proxy
17:44:39	kserve-ci-e2e-test	kubelet	error-404-isvc-61a23-predictor-fd5b4c6d4-52d4w	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:44:40	kserve-ci-e2e-test	kubelet	success-200-isvc-61a23-predictor-644f77f65c-7gb8k	Created	Created container: kube-rbac-proxy
17:44:40	kserve-ci-e2e-test	kubelet	success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj	Unhealthy	Readiness probe failed: Get "https://10.133.0.19:8643/healthz": dial tcp 10.133.0.19:8643: connect: connection refused
17:44:40	kserve-ci-e2e-test	kubelet	success-200-isvc-61a23-predictor-644f77f65c-7gb8k	Started	Started container kube-rbac-proxy
17:44:40 (x7)	kserve-ci-e2e-test	kubelet	success-200-isvc-b1dfe-predictor-559469fcc9-cjnnj	Unhealthy	Readiness probe failed: dial tcp 10.133.0.19:8080: connect: connection refused
17:44:40	kserve-ci-e2e-test	kubelet	success-200-isvc-61a23-predictor-644f77f65c-7gb8k	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:44:40	kserve-ci-e2e-test	kubelet	success-200-isvc-61a23-predictor-644f77f65c-7gb8k	Started	Started container kserve-container
17:44:40	kserve-ci-e2e-test	kubelet	success-200-isvc-61a23-predictor-644f77f65c-7gb8k	Created	Created container: kserve-container
17:44:40	kserve-ci-e2e-test	kubelet	success-200-isvc-61a23-predictor-644f77f65c-7gb8k	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1821" already present on machine
17:44:40	kserve-ci-e2e-test	multus	success-200-isvc-61a23-predictor-644f77f65c-7gb8k	AddedInterface	Add eth0 [10.133.0.23/23] from ovn-kubernetes
17:44:42	kserve-ci-e2e-test	kubelet	error-404-isvc-8b8b5-predictor-68f9fbd6b8-5djw5	Unhealthy	Readiness probe failed: Get "https://10.134.0.41:8643/healthz": dial tcp 10.134.0.41:8643: connect: connection refused
17:44:42	kserve-ci-e2e-test	kubelet	success-200-isvc-8b8b5-predictor-7f4bbc59f7-2hshk	Unhealthy	Readiness probe failed: Get "https://10.133.0.21:8643/healthz": dial tcp 10.133.0.21:8643: connect: connection refused
17:44:53	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-61a23-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:44:53	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-61a23-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:45:04 (x6)	kserve-ci-e2e-test	kubelet	sequence-graph-b1dfe-bf8f99978-2vj8v	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
17:45:05 (x6)	kserve-ci-e2e-test	kubelet	ensemble-graph-8b8b5-846dd8bd57-vd26j	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
17:45:08 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-61a23-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:45:08 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-614a5-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:45:08 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-614a5-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:45:08 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-61a23-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:45:09 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-614a5-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:45:09 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-614a5-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:45:15 (x6)	kserve-ci-e2e-test	kubelet	error-404-isvc-61a23-predictor-fd5b4c6d4-52d4w	Unhealthy	Readiness probe failed: dial tcp 10.134.0.42:8080: connect: connection refused
17:45:17 (x6)	kserve-ci-e2e-test	kubelet	success-200-isvc-61a23-predictor-644f77f65c-7gb8k	Unhealthy	Readiness probe failed: dial tcp 10.133.0.23:8080: connect: connection refused
17:45:25	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-614a5	InferenceServiceReady	InferenceService [success-200-isvc-614a5] is Ready
17:45:25 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-614a5	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:45:26	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-61a23	InferenceServiceReady	InferenceService [error-404-isvc-61a23] is Ready
17:45:26 (x5)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-61a23	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:45:26	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-614a5	InferenceServiceReady	InferenceService [error-404-isvc-614a5] is Ready
17:45:26 (x5)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-614a5	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:45:27 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-61a23	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:45:27	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-61a23	InferenceServiceReady	InferenceService [success-200-isvc-61a23] is Ready
17:45:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-61a23-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:45:38	kserve-ci-e2e-test	deployment-controller	ensemble-graph-614a5	ScalingReplicaSet	Scaled up replica set ensemble-graph-614a5-7dcdd75c from 0 to 1
17:45:38	kserve-ci-e2e-test	InferenceGraphController	ensemble-graph-614a5	InternalError	fails to update InferenceGraph status: Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "ensemble-graph-614a5": the object has been modified; please apply your changes to the latest version and try again
17:45:38	kserve-ci-e2e-test	InferenceGraphController	sequence-graph-61a23	InferenceGraphReady	InferenceGraph [sequence-graph-61a23] is Ready
17:45:38	kserve-ci-e2e-test	deployment-controller	sequence-graph-61a23	ScalingReplicaSet	Scaled up replica set sequence-graph-61a23-7dfbd4b7cf from 0 to 1
17:45:38	kserve-ci-e2e-test	InferenceGraphController	ensemble-graph-614a5	InferenceGraphReady	InferenceGraph [ensemble-graph-614a5] is Ready
17:45:38	kserve-ci-e2e-test	replicaset-controller	sequence-graph-61a23-7dfbd4b7cf	SuccessfulCreate	Created pod: sequence-graph-61a23-7dfbd4b7cf-ll569
17:45:38 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-61a23-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:45:38	kserve-ci-e2e-test	kubelet	sequence-graph-61a23-7dfbd4b7cf-ll569	Started	Started container sequence-graph-61a23
17:45:38	kserve-ci-e2e-test	replicaset-controller	ensemble-graph-614a5-7dcdd75c	SuccessfulCreate	Created pod: ensemble-graph-614a5-7dcdd75c-qs5jf
17:45:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-61a23-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:45:38	kserve-ci-e2e-test	InferenceGraphController	ensemble-graph-614a5	UpdateFailed	Failed to update status for InferenceGraph "ensemble-graph-614a5": Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "ensemble-graph-614a5": the object has been modified; please apply your changes to the latest version and try again
17:45:38 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-61a23-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:45:38	kserve-ci-e2e-test	kubelet	sequence-graph-61a23-7dfbd4b7cf-ll569	Created	Created container: sequence-graph-61a23
17:45:38	kserve-ci-e2e-test	kubelet	sequence-graph-61a23-7dfbd4b7cf-ll569	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:2d78b1e75ce4a3e633713c1f964bde1c2425164638b26ad77af20d6340c844d7" already present on machine
17:45:38	kserve-ci-e2e-test	multus	sequence-graph-61a23-7dfbd4b7cf-ll569	AddedInterface	Add eth0 [10.133.0.25/23] from ovn-kubernetes
17:45:39	kserve-ci-e2e-test	kubelet	ensemble-graph-614a5-7dcdd75c-qs5jf	Created	Created container: ensemble-graph-614a5
17:45:39	kserve-ci-e2e-test	multus	ensemble-graph-614a5-7dcdd75c-qs5jf	AddedInterface	Add eth0 [10.133.0.26/23] from ovn-kubernetes
17:45:39	kserve-ci-e2e-test	kubelet	ensemble-graph-614a5-7dcdd75c-qs5jf	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "ensemble-graph-614a5-serving-cert" not found
17:45:39	kserve-ci-e2e-test	kubelet	ensemble-graph-614a5-7dcdd75c-qs5jf	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:2d78b1e75ce4a3e633713c1f964bde1c2425164638b26ad77af20d6340c844d7" already present on machine
17:45:39	kserve-ci-e2e-test	kubelet	ensemble-graph-614a5-7dcdd75c-qs5jf	Started	Started container ensemble-graph-614a5
17:45:52	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-787a1	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-787a1": the object has been modified; please apply your changes to the latest version and try again
17:45:52	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-787a1-predictor-fcb464fdb	SuccessfulCreate	Created pod: error-404-isvc-787a1-predictor-fcb464fdb-zzl47
17:45:52	kserve-ci-e2e-test	deployment-controller	success-200-isvc-787a1-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-787a1-predictor-869dbdb759 from 0 to 1
17:45:52	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-787a1-predictor-869dbdb759	SuccessfulCreate	Created pod: success-200-isvc-787a1-predictor-869dbdb759-qnbjk
17:45:52	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-787a1	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-787a1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-787a1": the object has been modified; please apply your changes to the latest version and try again
17:45:52	kserve-ci-e2e-test	kubelet	success-200-isvc-787a1-predictor-869dbdb759-qnbjk	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "success-200-isvc-787a1-predictor-serving-cert" not found
17:45:52	kserve-ci-e2e-test	kubelet	error-404-isvc-61a23-predictor-fd5b4c6d4-52d4w	Killing	Stopping container kserve-container
17:45:52	kserve-ci-e2e-test	kubelet	error-404-isvc-61a23-predictor-fd5b4c6d4-52d4w	Killing	Stopping container kube-rbac-proxy
17:45:52	kserve-ci-e2e-test	kubelet	success-200-isvc-61a23-predictor-644f77f65c-7gb8k	Unhealthy	Readiness probe failed: Get "https://10.133.0.23:8643/healthz": dial tcp 10.133.0.23:8643: connect: connection refused
17:45:52	kserve-ci-e2e-test	kubelet	success-200-isvc-61a23-predictor-644f77f65c-7gb8k	Killing	Stopping container kube-rbac-proxy
17:45:52	kserve-ci-e2e-test	kubelet	success-200-isvc-61a23-predictor-644f77f65c-7gb8k	Killing	Stopping container kserve-container
17:45:52	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-787a1	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-787a1": the object has been modified; please apply your changes to the latest version and try again
17:45:52	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-787a1	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-787a1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-787a1": the object has been modified; please apply your changes to the latest version and try again
17:45:52	kserve-ci-e2e-test	kubelet	sequence-graph-61a23-7dfbd4b7cf-ll569	Killing	Stopping container sequence-graph-61a23
17:45:52	kserve-ci-e2e-test	deployment-controller	error-404-isvc-787a1-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-787a1-predictor-fcb464fdb from 0 to 1
17:45:53	kserve-ci-e2e-test	multus	success-200-isvc-787a1-predictor-869dbdb759-qnbjk	AddedInterface	Add eth0 [10.133.0.27/23] from ovn-kubernetes
17:45:53	kserve-ci-e2e-test	kubelet	success-200-isvc-787a1-predictor-869dbdb759-qnbjk	Created	Created container: kube-rbac-proxy
17:45:53	kserve-ci-e2e-test	kubelet	error-404-isvc-787a1-predictor-fcb464fdb-zzl47	Created	Created container: kserve-container
17:45:53	kserve-ci-e2e-test	multus	error-404-isvc-787a1-predictor-fcb464fdb-zzl47	AddedInterface	Add eth0 [10.134.0.44/23] from ovn-kubernetes
17:45:53 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-614a5-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:45:53	kserve-ci-e2e-test	kubelet	error-404-isvc-787a1-predictor-fcb464fdb-zzl47	Started	Started container kserve-container
17:45:53	kserve-ci-e2e-test	kubelet	error-404-isvc-787a1-predictor-fcb464fdb-zzl47	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:45:53 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-614a5-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:45:53	kserve-ci-e2e-test	kubelet	success-200-isvc-787a1-predictor-869dbdb759-qnbjk	Started	Started container kube-rbac-proxy
17:45:53	kserve-ci-e2e-test	kubelet	error-404-isvc-787a1-predictor-fcb464fdb-zzl47	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1821" already present on machine
17:45:53	kserve-ci-e2e-test	kubelet	success-200-isvc-787a1-predictor-869dbdb759-qnbjk	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:45:53	kserve-ci-e2e-test	kubelet	success-200-isvc-787a1-predictor-869dbdb759-qnbjk	Started	Started container kserve-container
17:45:53	kserve-ci-e2e-test	kubelet	success-200-isvc-787a1-predictor-869dbdb759-qnbjk	Created	Created container: kserve-container
17:45:53	kserve-ci-e2e-test	kubelet	error-404-isvc-787a1-predictor-fcb464fdb-zzl47	Started	Started container kube-rbac-proxy
17:45:53	kserve-ci-e2e-test	horizontal-pod-autoscaler	ensemble-graph-614a5	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:45:53	kserve-ci-e2e-test	horizontal-pod-autoscaler	ensemble-graph-614a5	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:45:53	kserve-ci-e2e-test	kubelet	success-200-isvc-787a1-predictor-869dbdb759-qnbjk	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1821" already present on machine
17:45:53	kserve-ci-e2e-test	kubelet	error-404-isvc-787a1-predictor-fcb464fdb-zzl47	Created	Created container: kube-rbac-proxy
17:45:54 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-614a5-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:45:54 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-614a5-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:46:08	kserve-ci-e2e-test	horizontal-pod-autoscaler	ensemble-graph-614a5	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:46:08	kserve-ci-e2e-test	horizontal-pod-autoscaler	ensemble-graph-614a5	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:46:20 (x6)	kserve-ci-e2e-test	kubelet	sequence-graph-61a23-7dfbd4b7cf-ll569	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
17:46:22 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-787a1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:46:22 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-787a1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:46:37 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-787a1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:46:37 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-787a1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:46:39 (x5)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-787a1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:46:39	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-787a1	InferenceServiceReady	InferenceService [error-404-isvc-787a1] is Ready
17:46:40 (x6)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-787a1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:46:40	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-787a1	InferenceServiceReady	InferenceService [success-200-isvc-787a1] is Ready
17:46:52	kserve-ci-e2e-test	replicaset-controller	sequence-graph-787a1-77c7b956dc	SuccessfulCreate	Created pod: sequence-graph-787a1-77c7b956dc-cpngn
17:46:52	kserve-ci-e2e-test	deployment-controller	sequence-graph-787a1	ScalingReplicaSet	Scaled up replica set sequence-graph-787a1-77c7b956dc from 0 to 1
17:46:52	kserve-ci-e2e-test	InferenceGraphController	sequence-graph-787a1	InferenceGraphReady	InferenceGraph [sequence-graph-787a1] is Ready
17:46:53	kserve-ci-e2e-test	multus	sequence-graph-787a1-77c7b956dc-cpngn	AddedInterface	Add eth0 [10.133.0.28/23] from ovn-kubernetes
17:46:53	kserve-ci-e2e-test	kubelet	sequence-graph-787a1-77c7b956dc-cpngn	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:2d78b1e75ce4a3e633713c1f964bde1c2425164638b26ad77af20d6340c844d7" already present on machine
17:46:53	kserve-ci-e2e-test	kubelet	sequence-graph-787a1-77c7b956dc-cpngn	Created	Created container: sequence-graph-787a1
17:46:53	kserve-ci-e2e-test	kubelet	sequence-graph-787a1-77c7b956dc-cpngn	Started	Started container sequence-graph-787a1
17:47:07 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-787a1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:47:07 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-787a1-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:47:07 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-787a1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:47:07 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-787a1-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:47:37 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-787a1	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:47:37 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-787a1	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:53:53	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-f7704	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-f7704": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-f7704": the object has been modified; please apply your changes to the latest version and try again
17:53:53	kserve-ci-e2e-test	deployment-controller	success-200-isvc-f7704-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-f7704-predictor-865dcf7dff from 0 to 1
17:53:53	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-f7704-predictor-865dcf7dff	SuccessfulCreate	Created pod: success-200-isvc-f7704-predictor-865dcf7dff-bjhft
17:53:53	kserve-ci-e2e-test	deployment-controller	error-404-isvc-f7704-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-f7704-predictor-5b75c47dd6 from 0 to 1
17:53:53	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-f7704	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-f7704": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-f7704": the object has been modified; please apply your changes to the latest version and try again
17:53:53	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-f7704	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-f7704": the object has been modified; please apply your changes to the latest version and try again
17:53:53	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-f7704-predictor-5b75c47dd6	SuccessfulCreate	Created pod: error-404-isvc-f7704-predictor-5b75c47dd6-jxdm4
17:53:53	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-f7704	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-f7704": the object has been modified; please apply your changes to the latest version and try again
17:53:53	kserve-ci-e2e-test	kubelet	error-404-isvc-614a5-predictor-55c87cb68d-g9s48	Killing	Stopping container kube-rbac-proxy
17:53:53	kserve-ci-e2e-test	kubelet	success-200-isvc-614a5-predictor-776c647d45-4xwq8	Killing	Stopping container kube-rbac-proxy
17:53:53	kserve-ci-e2e-test	kubelet	error-404-isvc-614a5-predictor-55c87cb68d-g9s48	Killing	Stopping container kserve-container
17:53:53	kserve-ci-e2e-test	kubelet	success-200-isvc-614a5-predictor-776c647d45-4xwq8	Killing	Stopping container kserve-container
17:53:53	kserve-ci-e2e-test	kubelet	ensemble-graph-614a5-7dcdd75c-qs5jf	Killing	Stopping container ensemble-graph-614a5
17:53:53	kserve-ci-e2e-test	kubelet	error-404-isvc-f7704-predictor-5b75c47dd6-jxdm4	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "error-404-isvc-f7704-predictor-serving-cert" not found
17:53:54	kserve-ci-e2e-test	kubelet	error-404-isvc-f7704-predictor-5b75c47dd6-jxdm4	Created	Created container: kserve-container
17:53:54	kserve-ci-e2e-test	kubelet	success-200-isvc-f7704-predictor-865dcf7dff-bjhft	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:53:54	kserve-ci-e2e-test	kubelet	error-404-isvc-f7704-predictor-5b75c47dd6-jxdm4	Started	Started container kube-rbac-proxy
17:53:54	kserve-ci-e2e-test	multus	error-404-isvc-f7704-predictor-5b75c47dd6-jxdm4	AddedInterface	Add eth0 [10.134.0.45/23] from ovn-kubernetes
17:53:54	kserve-ci-e2e-test	kubelet	error-404-isvc-f7704-predictor-5b75c47dd6-jxdm4	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1821" already present on machine
17:53:54	kserve-ci-e2e-test	kubelet	success-200-isvc-f7704-predictor-865dcf7dff-bjhft	Created	Created container: kserve-container
17:53:54	kserve-ci-e2e-test	kubelet	success-200-isvc-f7704-predictor-865dcf7dff-bjhft	Created	Created container: kube-rbac-proxy
17:53:54	kserve-ci-e2e-test	kubelet	error-404-isvc-f7704-predictor-5b75c47dd6-jxdm4	Created	Created container: kube-rbac-proxy
17:53:54	kserve-ci-e2e-test	kubelet	success-200-isvc-f7704-predictor-865dcf7dff-bjhft	Started	Started container kube-rbac-proxy
17:53:54	kserve-ci-e2e-test	kubelet	success-200-isvc-f7704-predictor-865dcf7dff-bjhft	Started	Started container kserve-container
17:53:54	kserve-ci-e2e-test	kubelet	error-404-isvc-f7704-predictor-5b75c47dd6-jxdm4	Started	Started container kserve-container
17:53:54	kserve-ci-e2e-test	kubelet	success-200-isvc-f7704-predictor-865dcf7dff-bjhft	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1821" already present on machine
17:53:54	kserve-ci-e2e-test	kubelet	error-404-isvc-f7704-predictor-5b75c47dd6-jxdm4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:53:54	kserve-ci-e2e-test	multus	success-200-isvc-f7704-predictor-865dcf7dff-bjhft	AddedInterface	Add eth0 [10.133.0.29/23] from ovn-kubernetes
17:53:55	kserve-ci-e2e-test	kubelet	success-200-isvc-614a5-predictor-776c647d45-4xwq8	Unhealthy	Readiness probe failed: Get "https://10.133.0.24:8643/healthz": dial tcp 10.133.0.24:8643: connect: connection refused
17:53:55 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-614a5-predictor-55c87cb68d-g9s48	Unhealthy	Readiness probe failed: dial tcp 10.134.0.43:8080: connect: connection refused
17:53:55	kserve-ci-e2e-test	kubelet	error-404-isvc-614a5-predictor-55c87cb68d-g9s48	Unhealthy	Readiness probe failed: Get "https://10.134.0.43:8643/healthz": dial tcp 10.134.0.43:8643: connect: connection refused
17:53:55 (x7)	kserve-ci-e2e-test	kubelet	success-200-isvc-614a5-predictor-776c647d45-4xwq8	Unhealthy	Readiness probe failed: dial tcp 10.133.0.24:8080: connect: connection refused
17:54:08	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-f7704-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:54:08	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-f7704-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:54:20 (x6)	kserve-ci-e2e-test	kubelet	ensemble-graph-614a5-7dcdd75c-qs5jf	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
17:54:31 (x6)	kserve-ci-e2e-test	kubelet	success-200-isvc-f7704-predictor-865dcf7dff-bjhft	Unhealthy	Readiness probe failed: dial tcp 10.133.0.29:8080: connect: connection refused
17:54:32 (x6)	kserve-ci-e2e-test	kubelet	error-404-isvc-f7704-predictor-5b75c47dd6-jxdm4	Unhealthy	Readiness probe failed: dial tcp 10.134.0.45:8080: connect: connection refused
17:54:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-f7704-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:54:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-f7704-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:54:41	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-f7704	InferenceServiceReady	InferenceService [success-200-isvc-f7704] is Ready
17:54:41 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-f7704	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:54:42	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-f7704	InferenceServiceReady	InferenceService [error-404-isvc-f7704] is Ready
17:54:42 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-f7704	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:54:53	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-f7704-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:54:53	kserve-ci-e2e-test	replicaset-controller	splitter-graph-f7704-758c568bf	SuccessfulCreate	Created pod: splitter-graph-f7704-758c568bf-l924l
17:54:53	kserve-ci-e2e-test	deployment-controller	splitter-graph-f7704	ScalingReplicaSet	Scaled up replica set splitter-graph-f7704-758c568bf from 0 to 1
17:54:53	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-f7704-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:54:53	kserve-ci-e2e-test	kubelet	splitter-graph-f7704-758c568bf-l924l	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "splitter-graph-f7704-serving-cert" not found
17:54:53 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-f7704-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:54:53 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-f7704-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:54:53	kserve-ci-e2e-test	InferenceGraphController	splitter-graph-f7704	InferenceGraphReady	InferenceGraph [splitter-graph-f7704] is Ready
17:54:54	kserve-ci-e2e-test	kubelet	splitter-graph-f7704-758c568bf-l924l	Started	Started container splitter-graph-f7704
17:54:54	kserve-ci-e2e-test	kubelet	splitter-graph-f7704-758c568bf-l924l	Created	Created container: splitter-graph-f7704
17:54:54	kserve-ci-e2e-test	multus	splitter-graph-f7704-758c568bf-l924l	AddedInterface	Add eth0 [10.133.0.30/23] from ovn-kubernetes
17:54:54	kserve-ci-e2e-test	kubelet	splitter-graph-f7704-758c568bf-l924l	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:2d78b1e75ce4a3e633713c1f964bde1c2425164638b26ad77af20d6340c844d7" already present on machine
17:55:03	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-7d999	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-7d999": the object has been modified; please apply your changes to the latest version and try again
17:55:03	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-7d999	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-7d999": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-7d999": the object has been modified; please apply your changes to the latest version and try again
17:55:03	kserve-ci-e2e-test	deployment-controller	success-200-isvc-7d999-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-7d999-predictor-7f48846ddc from 0 to 1
17:55:03	kserve-ci-e2e-test	kubelet	splitter-graph-f7704-758c568bf-l924l	Killing	Stopping container splitter-graph-f7704
17:55:04	kserve-ci-e2e-test	kubelet	error-404-isvc-7d999-predictor-566585ff9d-rtwzr	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "error-404-isvc-7d999-predictor-serving-cert" not found
17:55:04	kserve-ci-e2e-test	deployment-controller	error-404-isvc-7d999-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-7d999-predictor-566585ff9d from 0 to 1
17:55:04	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-7d999-predictor-7f48846ddc	SuccessfulCreate	Created pod: success-200-isvc-7d999-predictor-7f48846ddc-d5stl
17:55:04	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-7d999-predictor-566585ff9d	SuccessfulCreate	Created pod: error-404-isvc-7d999-predictor-566585ff9d-rtwzr
17:55:04	kserve-ci-e2e-test	kubelet	success-200-isvc-f7704-predictor-865dcf7dff-bjhft	Killing	Stopping container kube-rbac-proxy
17:55:04	kserve-ci-e2e-test	kubelet	error-404-isvc-f7704-predictor-5b75c47dd6-jxdm4	Killing	Stopping container kserve-container
17:55:04	kserve-ci-e2e-test	kubelet	error-404-isvc-f7704-predictor-5b75c47dd6-jxdm4	Killing	Stopping container kube-rbac-proxy
17:55:04	kserve-ci-e2e-test	kubelet	success-200-isvc-7d999-predictor-7f48846ddc-d5stl	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "success-200-isvc-7d999-predictor-serving-cert" not found
17:55:04	kserve-ci-e2e-test	kubelet	success-200-isvc-f7704-predictor-865dcf7dff-bjhft	Killing	Stopping container kserve-container
17:55:05	kserve-ci-e2e-test	kubelet	success-200-isvc-7d999-predictor-7f48846ddc-d5stl	Created	Created container: kube-rbac-proxy
17:55:05	kserve-ci-e2e-test	kubelet	error-404-isvc-7d999-predictor-566585ff9d-rtwzr	Started	Started container kube-rbac-proxy
17:55:05	kserve-ci-e2e-test	kubelet	error-404-isvc-7d999-predictor-566585ff9d-rtwzr	Created	Created container: kube-rbac-proxy
17:55:05	kserve-ci-e2e-test	kubelet	error-404-isvc-7d999-predictor-566585ff9d-rtwzr	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:55:05	kserve-ci-e2e-test	kubelet	error-404-isvc-7d999-predictor-566585ff9d-rtwzr	Started	Started container kserve-container
17:55:05	kserve-ci-e2e-test	kubelet	error-404-isvc-7d999-predictor-566585ff9d-rtwzr	Created	Created container: kserve-container
17:55:05	kserve-ci-e2e-test	kubelet	error-404-isvc-7d999-predictor-566585ff9d-rtwzr	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1821" already present on machine
17:55:05	kserve-ci-e2e-test	multus	error-404-isvc-7d999-predictor-566585ff9d-rtwzr	AddedInterface	Add eth0 [10.134.0.46/23] from ovn-kubernetes
17:55:05	kserve-ci-e2e-test	kubelet	success-200-isvc-7d999-predictor-7f48846ddc-d5stl	Started	Started container kube-rbac-proxy
17:55:05	kserve-ci-e2e-test	kubelet	success-200-isvc-7d999-predictor-7f48846ddc-d5stl	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:55:05	kserve-ci-e2e-test	kubelet	success-200-isvc-7d999-predictor-7f48846ddc-d5stl	Started	Started container kserve-container
17:55:05	kserve-ci-e2e-test	kubelet	success-200-isvc-7d999-predictor-7f48846ddc-d5stl	Created	Created container: kserve-container
17:55:05	kserve-ci-e2e-test	kubelet	success-200-isvc-7d999-predictor-7f48846ddc-d5stl	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1821" already present on machine
17:55:05	kserve-ci-e2e-test	multus	success-200-isvc-7d999-predictor-7f48846ddc-d5stl	AddedInterface	Add eth0 [10.133.0.31/23] from ovn-kubernetes
17:55:06	kserve-ci-e2e-test	kubelet	success-200-isvc-f7704-predictor-865dcf7dff-bjhft	Unhealthy	Readiness probe failed: Get "https://10.133.0.29:8643/healthz": dial tcp 10.133.0.29:8643: connect: connection refused
17:55:07	kserve-ci-e2e-test	kubelet	sequence-graph-787a1-77c7b956dc-cpngn	Killing	Stopping container sequence-graph-787a1
17:55:07	kserve-ci-e2e-test	deployment-controller	error-404-isvc-cb1ee-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-cb1ee-predictor-8494ff7d66 from 0 to 1
17:55:07	kserve-ci-e2e-test	kubelet	success-200-isvc-787a1-predictor-869dbdb759-qnbjk	Killing	Stopping container kube-rbac-proxy
17:55:07	kserve-ci-e2e-test	kubelet	success-200-isvc-787a1-predictor-869dbdb759-qnbjk	Killing	Stopping container kserve-container
17:55:07	kserve-ci-e2e-test	kubelet	error-404-isvc-787a1-predictor-fcb464fdb-zzl47	Killing	Stopping container kserve-container
17:55:07	kserve-ci-e2e-test	kubelet	error-404-isvc-787a1-predictor-fcb464fdb-zzl47	Killing	Stopping container kube-rbac-proxy
17:55:07	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-cb1ee	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-cb1ee": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-cb1ee": the object has been modified; please apply your changes to the latest version and try again
17:55:07	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-cb1ee	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-cb1ee": the object has been modified; please apply your changes to the latest version and try again
17:55:07	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-cb1ee-predictor-8494ff7d66	SuccessfulCreate	Created pod: error-404-isvc-cb1ee-predictor-8494ff7d66-sv48x
17:55:07	kserve-ci-e2e-test	deployment-controller	success-200-isvc-cb1ee-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-cb1ee-predictor-7b757b875f from 0 to 1
17:55:07	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-cb1ee-predictor-7b757b875f	SuccessfulCreate	Created pod: success-200-isvc-cb1ee-predictor-7b757b875f-46rjc
17:55:07	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-cb1ee	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-cb1ee": the object has been modified; please apply your changes to the latest version and try again
17:55:07	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-cb1ee	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-cb1ee": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-cb1ee": the object has been modified; please apply your changes to the latest version and try again
17:55:07	kserve-ci-e2e-test	kubelet	success-200-isvc-cb1ee-predictor-7b757b875f-46rjc	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "success-200-isvc-cb1ee-predictor-serving-cert" not found
17:55:08	kserve-ci-e2e-test	kubelet	success-200-isvc-cb1ee-predictor-7b757b875f-46rjc	Created	Created container: kube-rbac-proxy
17:55:08	kserve-ci-e2e-test	kubelet	error-404-isvc-cb1ee-predictor-8494ff7d66-sv48x	Created	Created container: kserve-container
17:55:08	kserve-ci-e2e-test	kubelet	success-200-isvc-cb1ee-predictor-7b757b875f-46rjc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:55:08	kserve-ci-e2e-test	kubelet	success-200-isvc-cb1ee-predictor-7b757b875f-46rjc	Started	Started container kserve-container
17:55:08	kserve-ci-e2e-test	kubelet	success-200-isvc-cb1ee-predictor-7b757b875f-46rjc	Created	Created container: kserve-container
17:55:08	kserve-ci-e2e-test	kubelet	success-200-isvc-cb1ee-predictor-7b757b875f-46rjc	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1821" already present on machine
17:55:08	kserve-ci-e2e-test	multus	success-200-isvc-cb1ee-predictor-7b757b875f-46rjc	AddedInterface	Add eth0 [10.133.0.32/23] from ovn-kubernetes
17:55:08	kserve-ci-e2e-test	kubelet	success-200-isvc-cb1ee-predictor-7b757b875f-46rjc	Started	Started container kube-rbac-proxy
17:55:08	kserve-ci-e2e-test	kubelet	error-404-isvc-cb1ee-predictor-8494ff7d66-sv48x	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1821" already present on machine
17:55:08	kserve-ci-e2e-test	multus	error-404-isvc-cb1ee-predictor-8494ff7d66-sv48x	AddedInterface	Add eth0 [10.134.0.47/23] from ovn-kubernetes
17:55:08	kserve-ci-e2e-test	kubelet	error-404-isvc-cb1ee-predictor-8494ff7d66-sv48x	Started	Started container kserve-container
17:55:08	kserve-ci-e2e-test	kubelet	error-404-isvc-cb1ee-predictor-8494ff7d66-sv48x	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
17:55:08	kserve-ci-e2e-test	kubelet	error-404-isvc-cb1ee-predictor-8494ff7d66-sv48x	Created	Created container: kube-rbac-proxy
17:55:08	kserve-ci-e2e-test	kubelet	error-404-isvc-f7704-predictor-5b75c47dd6-jxdm4	Unhealthy	Readiness probe failed: Get "https://10.134.0.45:8643/healthz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
17:55:08	kserve-ci-e2e-test	kubelet	error-404-isvc-cb1ee-predictor-8494ff7d66-sv48x	Started	Started container kube-rbac-proxy
17:55:09 (x7)	kserve-ci-e2e-test	kubelet	success-200-isvc-787a1-predictor-869dbdb759-qnbjk	Unhealthy	Readiness probe failed: dial tcp 10.133.0.27:8080: connect: connection refused
17:55:09	kserve-ci-e2e-test	kubelet	error-404-isvc-787a1-predictor-fcb464fdb-zzl47	Unhealthy	Readiness probe failed: Get "https://10.134.0.44:8643/healthz": dial tcp 10.134.0.44:8643: connect: connection refused
17:55:09	kserve-ci-e2e-test	kubelet	success-200-isvc-787a1-predictor-869dbdb759-qnbjk	Unhealthy	Readiness probe failed: Get "https://10.133.0.27:8643/healthz": dial tcp 10.133.0.27:8643: connect: connection refused
17:55:09 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-787a1-predictor-fcb464fdb-zzl47	Unhealthy	Readiness probe failed: dial tcp 10.134.0.44:8080: connect: connection refused
17:55:19	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-7d999-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:55:19	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-7d999-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:55:30 (x6)	kserve-ci-e2e-test	kubelet	splitter-graph-f7704-758c568bf-l924l	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
17:55:34 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-7d999-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:55:34 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-7d999-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:55:34 (x6)	kserve-ci-e2e-test	kubelet	sequence-graph-787a1-77c7b956dc-cpngn	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
17:55:37 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-cb1ee-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:55:37 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-cb1ee-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:55:41 (x6)	kserve-ci-e2e-test	kubelet	success-200-isvc-7d999-predictor-7f48846ddc-d5stl	Unhealthy	Readiness probe failed: dial tcp 10.133.0.31:8080: connect: connection refused
17:55:42 (x6)	kserve-ci-e2e-test	kubelet	error-404-isvc-7d999-predictor-566585ff9d-rtwzr	Unhealthy	Readiness probe failed: dial tcp 10.134.0.46:8080: connect: connection refused
17:55:45 (x6)	kserve-ci-e2e-test	kubelet	error-404-isvc-cb1ee-predictor-8494ff7d66-sv48x	Unhealthy	Readiness probe failed: dial tcp 10.134.0.47:8080: connect: connection refused
17:55:52 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-7d999	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:55:52	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-7d999	InferenceServiceReady	InferenceService [success-200-isvc-7d999] is Ready
17:55:52 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-cb1ee-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:55:52 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-cb1ee-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:55:53	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-7d999	InferenceServiceReady	InferenceService [error-404-isvc-7d999] is Ready
17:55:53 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-7d999	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:55:55 (x7)	kserve-ci-e2e-test	kubelet	success-200-isvc-cb1ee-predictor-7b757b875f-46rjc	Unhealthy	Readiness probe failed: dial tcp 10.133.0.32:8080: connect: connection refused
17:55:56	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-cb1ee	InferenceServiceReady	InferenceService [error-404-isvc-cb1ee] is Ready
17:55:56 (x5)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-cb1ee	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:56:04	kserve-ci-e2e-test	kubelet	splitter-graph-7d999-849c5688bd-rrgg7	Created	Created container: splitter-graph-7d999
17:56:04	kserve-ci-e2e-test	InferenceGraphController	splitter-graph-7d999	InferenceGraphReady	InferenceGraph [splitter-graph-7d999] is Ready
17:56:04	kserve-ci-e2e-test	kubelet	splitter-graph-7d999-849c5688bd-rrgg7	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:2d78b1e75ce4a3e633713c1f964bde1c2425164638b26ad77af20d6340c844d7" already present on machine
17:56:04	kserve-ci-e2e-test	InferenceGraphController	splitter-graph-7d999	InternalError	fails to update InferenceGraph status: Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "splitter-graph-7d999": the object has been modified; please apply your changes to the latest version and try again
17:56:04	kserve-ci-e2e-test	kubelet	splitter-graph-7d999-849c5688bd-rrgg7	Started	Started container splitter-graph-7d999
17:56:04	kserve-ci-e2e-test	multus	splitter-graph-7d999-849c5688bd-rrgg7	AddedInterface	Add eth0 [10.133.0.33/23] from ovn-kubernetes
17:56:04 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-7d999-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:56:04 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-7d999-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:56:04	kserve-ci-e2e-test	InferenceGraphController	splitter-graph-7d999	UpdateFailed	Failed to update status for InferenceGraph "splitter-graph-7d999": Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "splitter-graph-7d999": the object has been modified; please apply your changes to the latest version and try again
17:56:04	kserve-ci-e2e-test	replicaset-controller	splitter-graph-7d999-849c5688bd	SuccessfulCreate	Created pod: splitter-graph-7d999-849c5688bd-rrgg7
17:56:04	kserve-ci-e2e-test	deployment-controller	splitter-graph-7d999	ScalingReplicaSet	Scaled up replica set splitter-graph-7d999-849c5688bd from 0 to 1
17:56:05	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-cb1ee	InferenceServiceReady	InferenceService [success-200-isvc-cb1ee] is Ready
17:56:05 (x6)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-cb1ee	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
17:56:17	kserve-ci-e2e-test	deployment-controller	switch-graph-cb1ee	ScalingReplicaSet	Scaled up replica set switch-graph-cb1ee-7ddbb59689 from 0 to 1
17:56:17	kserve-ci-e2e-test	replicaset-controller	switch-graph-cb1ee-7ddbb59689	SuccessfulCreate	Created pod: switch-graph-cb1ee-7ddbb59689-cbvqc
17:56:17	kserve-ci-e2e-test	kubelet	switch-graph-cb1ee-7ddbb59689-cbvqc	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "switch-graph-cb1ee-serving-cert" not found
17:56:17	kserve-ci-e2e-test	InferenceGraphController	switch-graph-cb1ee	InferenceGraphReady	InferenceGraph [switch-graph-cb1ee] is Ready
17:56:18	kserve-ci-e2e-test	multus	switch-graph-cb1ee-7ddbb59689-cbvqc	AddedInterface	Add eth0 [10.133.0.34/23] from ovn-kubernetes
17:56:18	kserve-ci-e2e-test	kubelet	switch-graph-cb1ee-7ddbb59689-cbvqc	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:2d78b1e75ce4a3e633713c1f964bde1c2425164638b26ad77af20d6340c844d7" already present on machine
17:56:18	kserve-ci-e2e-test	kubelet	switch-graph-cb1ee-7ddbb59689-cbvqc	Started	Started container switch-graph-cb1ee
17:56:18	kserve-ci-e2e-test	kubelet	switch-graph-cb1ee-7ddbb59689-cbvqc	Created	Created container: switch-graph-cb1ee
17:56:19 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-7d999-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:56:19 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-7d999-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:56:22 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-cb1ee-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:56:22 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-cb1ee-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:56:22 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-cb1ee-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:56:22 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-cb1ee-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:56:32	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-cb1ee	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:56:32	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-cb1ee	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:56:34 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	splitter-graph-7d999	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:56:34 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	splitter-graph-7d999	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
17:56:47	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-cb1ee	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
17:56:47	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-cb1ee	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
18:04:18	kserve-ci-e2e-test	kubelet	success-200-isvc-7d999-predictor-7f48846ddc-d5stl	Killing	Stopping container kube-rbac-proxy
18:04:18	kserve-ci-e2e-test	kubelet	success-200-isvc-7d999-predictor-7f48846ddc-d5stl	Killing	Stopping container kserve-container
18:04:18	kserve-ci-e2e-test	kubelet	splitter-graph-7d999-849c5688bd-rrgg7	Killing	Stopping container splitter-graph-7d999
18:04:18	kserve-ci-e2e-test	kubelet	error-404-isvc-7d999-predictor-566585ff9d-rtwzr	Killing	Stopping container kserve-container
18:04:18	kserve-ci-e2e-test	kubelet	error-404-isvc-7d999-predictor-566585ff9d-rtwzr	Killing	Stopping container kube-rbac-proxy
18:04:46 (x6)	kserve-ci-e2e-test	kubelet	splitter-graph-7d999-849c5688bd-rrgg7	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
18:12:36	kserve-ci-e2e-test	kubelet	error-404-isvc-cb1ee-predictor-8494ff7d66-sv48x	Killing	Stopping container kube-rbac-proxy
18:12:36	kserve-ci-e2e-test	kubelet	error-404-isvc-cb1ee-predictor-8494ff7d66-sv48x	Killing	Stopping container kserve-container
18:12:36	kserve-ci-e2e-test	kubelet	switch-graph-cb1ee-7ddbb59689-cbvqc	Killing	Stopping container switch-graph-cb1ee
18:12:36	kserve-ci-e2e-test	kubelet	success-200-isvc-cb1ee-predictor-7b757b875f-46rjc	Killing	Stopping container kserve-container
18:12:36	kserve-ci-e2e-test	kubelet	success-200-isvc-cb1ee-predictor-7b757b875f-46rjc	Killing	Stopping container kube-rbac-proxy
18:13:05 (x6)	kserve-ci-e2e-test	kubelet	switch-graph-cb1ee-7ddbb59689-cbvqc	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503