Events

Time	Namespace	Component	RelatedObject	Reason	Message
10:54:43	kserve-ci-e2e-test		error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g to ip-10-0-131-34.ec2.internal
11:16:23	kserve-ci-e2e-test		success-200-isvc-cdb7a-predictor-54d956f49b-dzphd	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-cdb7a-predictor-54d956f49b-dzphd to ip-10-0-131-34.ec2.internal
11:18:33	kserve-ci-e2e-test		splitter-graph-d1ffd-7d9bf8b7bd-962wv	Scheduled	Successfully assigned kserve-ci-e2e-test/splitter-graph-d1ffd-7d9bf8b7bd-962wv to ip-10-0-130-140.ec2.internal
11:16:23	kserve-ci-e2e-test		error-404-isvc-cdb7a-predictor-66db674dcc-48hvs	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-cdb7a-predictor-66db674dcc-48hvs to ip-10-0-131-34.ec2.internal
11:17:23	kserve-ci-e2e-test		splitter-graph-cdb7a-ffd5b4495-dstlk	Scheduled	Successfully assigned kserve-ci-e2e-test/splitter-graph-cdb7a-ffd5b4495-dstlk to ip-10-0-130-140.ec2.internal
11:08:48	kserve-ci-e2e-test		sequence-graph-70137-5dcc4d4f8-8llvh	Scheduled	Successfully assigned kserve-ci-e2e-test/sequence-graph-70137-5dcc4d4f8-8llvh to ip-10-0-130-140.ec2.internal
11:07:38	kserve-ci-e2e-test		sequence-graph-30120-7b65c6b7f-q2rbc	Scheduled	Successfully assigned kserve-ci-e2e-test/sequence-graph-30120-7b65c6b7f-q2rbc to ip-10-0-130-140.ec2.internal
11:05:48	kserve-ci-e2e-test		error-404-isvc-cf436-predictor-777f9bff5f-j9df6	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-cf436-predictor-777f9bff5f-j9df6 to ip-10-0-131-34.ec2.internal
11:06:28	kserve-ci-e2e-test		success-200-isvc-30120-predictor-74997b5b5d-6hgnc	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-30120-predictor-74997b5b5d-6hgnc to ip-10-0-131-34.ec2.internal
10:54:42	kserve-ci-e2e-test		success-200-isvc-6db9e-predictor-58b664446b-fmthr	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-6db9e-predictor-58b664446b-fmthr to ip-10-0-131-34.ec2.internal
10:58:13	kserve-ci-e2e-test		sequence-graph-074d2-574bc49b98-k4s78	Scheduled	Successfully assigned kserve-ci-e2e-test/sequence-graph-074d2-574bc49b98-k4s78 to ip-10-0-130-140.ec2.internal
11:08:08	kserve-ci-e2e-test		ensemble-graph-d6666-6bd466db88-jkkg7	Scheduled	Successfully assigned kserve-ci-e2e-test/ensemble-graph-d6666-6bd466db88-jkkg7 to ip-10-0-130-140.ec2.internal
10:57:33	kserve-ci-e2e-test		switch-graph-ff4f8-fc4884fc4-c9572	Scheduled	Successfully assigned kserve-ci-e2e-test/switch-graph-ff4f8-fc4884fc4-c9572 to ip-10-0-130-140.ec2.internal
11:18:03	kserve-ci-e2e-test		switch-graph-bc774-7778877fb9-g5794	Scheduled	Successfully assigned kserve-ci-e2e-test/switch-graph-bc774-7778877fb9-g5794 to ip-10-0-130-140.ec2.internal
11:06:58	kserve-ci-e2e-test		ensemble-graph-cf436-7d4dbbfd97-vqcsn	Scheduled	Successfully assigned kserve-ci-e2e-test/ensemble-graph-cf436-7d4dbbfd97-vqcsn to ip-10-0-130-140.ec2.internal
10:57:02	kserve-ci-e2e-test		model-chainer-7fdbd8cf64-8vlgf	Scheduled	Successfully assigned kserve-ci-e2e-test/model-chainer-7fdbd8cf64-8vlgf to ip-10-0-130-140.ec2.internal
11:17:03	kserve-ci-e2e-test		error-404-isvc-bc774-predictor-8547545bdf-ktmgb	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-bc774-predictor-8547545bdf-ktmgb to ip-10-0-131-34.ec2.internal
11:07:48	kserve-ci-e2e-test		success-200-isvc-70137-predictor-77585cb866-kcdrt	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-70137-predictor-77585cb866-kcdrt to ip-10-0-131-34.ec2.internal
11:17:03	kserve-ci-e2e-test		success-200-isvc-bc774-predictor-5b6b775cdd-29slw	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-bc774-predictor-5b6b775cdd-29slw to ip-10-0-131-34.ec2.internal
11:05:47	kserve-ci-e2e-test		success-200-isvc-cf436-predictor-69cbcd7974-z8klj	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-cf436-predictor-69cbcd7974-z8klj to ip-10-0-131-34.ec2.internal
11:17:33	kserve-ci-e2e-test		error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2 to ip-10-0-131-34.ec2.internal
11:17:33	kserve-ci-e2e-test		success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf to ip-10-0-131-34.ec2.internal
11:07:08	kserve-ci-e2e-test		success-200-isvc-d6666-predictor-8674b65f5d-8hxmp	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-d6666-predictor-8674b65f5d-8hxmp to ip-10-0-131-34.ec2.internal
10:57:13	kserve-ci-e2e-test		error-404-isvc-074d2-predictor-576646885c-d8cfp	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-074d2-predictor-576646885c-d8cfp to ip-10-0-131-34.ec2.internal
11:07:48	kserve-ci-e2e-test		error-404-isvc-70137-predictor-544d7497cf-k6qvv	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-70137-predictor-544d7497cf-k6qvv to ip-10-0-131-34.ec2.internal
10:56:33	kserve-ci-e2e-test		success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9 to ip-10-0-131-34.ec2.internal
10:54:43	kserve-ci-e2e-test		isvc-xgboost-graph-predictor-555fc69d56-j57gn	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-j57gn to ip-10-0-131-34.ec2.internal
11:07:08	kserve-ci-e2e-test		error-404-isvc-d6666-predictor-774b85585f-t7hr4	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-d6666-predictor-774b85585f-t7hr4 to ip-10-0-131-34.ec2.internal
10:57:13	kserve-ci-e2e-test		success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp to ip-10-0-131-34.ec2.internal
10:54:43	kserve-ci-e2e-test		isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb to ip-10-0-131-34.ec2.internal
10:54:43	kserve-ci-e2e-test		isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh to ip-10-0-131-34.ec2.internal
11:06:28	kserve-ci-e2e-test		error-404-isvc-30120-predictor-6c4f57456c-6h8b6	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-30120-predictor-6c4f57456c-6h8b6 to ip-10-0-131-34.ec2.internal
10:56:33	kserve-ci-e2e-test		error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg to ip-10-0-131-34.ec2.internal
10:56:22	kserve-ci-e2e-test		switch-graph-6db9e-5cb89b98dc-7x78t	Scheduled	Successfully assigned kserve-ci-e2e-test/switch-graph-6db9e-5cb89b98dc-7x78t to ip-10-0-130-140.ec2.internal
10:54:42	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-6db9e	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-6db9e": the object has been modified; please apply your changes to the latest version and try again
10:54:42	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-6db9e-predictor-58b664446b	SuccessfulCreate	Created pod: success-200-isvc-6db9e-predictor-58b664446b-fmthr
10:54:42	kserve-ci-e2e-test	deployment-controller	success-200-isvc-6db9e-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-6db9e-predictor-58b664446b from 0 to 1
10:54:42	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-6db9e	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-6db9e": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-6db9e": the object has been modified; please apply your changes to the latest version and try again
10:54:43	kserve-ci-e2e-test	deployment-controller	error-404-isvc-6db9e-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-6db9e-predictor-6b44c644f9 from 0 to 1
10:54:43	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-graph-1-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-graph-1-predictor-5cb7bb944 from 0 to 1
10:54:43	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-graph-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-graph-predictor-555fc69d56 from 0 to 1
10:54:43	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-6db9e-predictor-6b44c644f9	SuccessfulCreate	Created pod: error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g
10:54:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
10:54:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb	Started	Started container storage-initializer
10:54:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb	Created	Created container: storage-initializer
10:54:43	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-graph-predictor-555fc69d56	SuccessfulCreate	Created pod: isvc-xgboost-graph-predictor-555fc69d56-j57gn
10:54:43	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-graph-1-predictor-5cb7bb944	SuccessfulCreate	Created pod: isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb
10:54:43	kserve-ci-e2e-test	kubelet	error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "error-404-isvc-6db9e-predictor-serving-cert" not found
10:54:43	kserve-ci-e2e-test	multus	isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb	AddedInterface	Add eth0 [10.133.0.22/23] from ovn-kubernetes
10:54:43	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-graph-2-predictor-d8594dbb4	SuccessfulCreate	Created pod: isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh
10:54:43	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-graph-2-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-graph-2-predictor-d8594dbb4 from 0 to 1
10:54:43	kserve-ci-e2e-test	multus	success-200-isvc-6db9e-predictor-58b664446b-fmthr	AddedInterface	Add eth0 [10.133.0.21/23] from ovn-kubernetes
10:54:43	kserve-ci-e2e-test	kubelet	success-200-isvc-6db9e-predictor-58b664446b-fmthr	Pulling	Pulling image "quay.io/opendatahub/success-200-isvc:odh-pr-1738"
10:54:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
10:54:44	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-j57gn	Started	Started container storage-initializer
10:54:44	kserve-ci-e2e-test	multus	error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g	AddedInterface	Add eth0 [10.133.0.23/23] from ovn-kubernetes
10:54:44	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-j57gn	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
10:54:44	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-j57gn	Created	Created container: storage-initializer
10:54:44	kserve-ci-e2e-test	multus	isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh	AddedInterface	Add eth0 [10.133.0.25/23] from ovn-kubernetes
10:54:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:523ed64da171f96ba586fa05d2bcb415c8c54dddf273afd4079bdc377a22c2e7" already present on machine
10:54:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh	Created	Created container: storage-initializer
10:54:44	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh	Started	Started container storage-initializer
10:54:44	kserve-ci-e2e-test	multus	isvc-xgboost-graph-predictor-555fc69d56-j57gn	AddedInterface	Add eth0 [10.133.0.24/23] from ovn-kubernetes
10:54:44	kserve-ci-e2e-test	kubelet	error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g	Pulling	Pulling image "quay.io/opendatahub/error-404-isvc:odh-pr-1738"
10:54:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1738"
10:54:50	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-j57gn	Pulling	Pulling image "kserve/xgbserver:latest"
10:54:50	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1738"
10:55:02	kserve-ci-e2e-test	kubelet	error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
10:55:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh	Created	Created container: kserve-container
10:55:02	kserve-ci-e2e-test	kubelet	success-200-isvc-6db9e-predictor-58b664446b-fmthr	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
10:55:02	kserve-ci-e2e-test	kubelet	error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g	Pulled	Successfully pulled image "quay.io/opendatahub/error-404-isvc:odh-pr-1738" in 17.714s (17.714s including waiting). Image size: 1334629472 bytes.
10:55:02	kserve-ci-e2e-test	kubelet	error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g	Created	Created container: kserve-container
10:55:02	kserve-ci-e2e-test	kubelet	error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g	Started	Started container kserve-container
10:55:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
10:55:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
10:55:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1738" in 11.441s (11.441s including waiting). Image size: 1555868070 bytes.
10:55:02	kserve-ci-e2e-test	kubelet	success-200-isvc-6db9e-predictor-58b664446b-fmthr	Started	Started container kserve-container
10:55:02	kserve-ci-e2e-test	kubelet	success-200-isvc-6db9e-predictor-58b664446b-fmthr	Created	Created container: kserve-container
10:55:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1738" in 11.454s (11.454s including waiting). Image size: 1555868070 bytes.
10:55:02	kserve-ci-e2e-test	kubelet	success-200-isvc-6db9e-predictor-58b664446b-fmthr	Pulled	Successfully pulled image "quay.io/opendatahub/success-200-isvc:odh-pr-1738" in 18.934s (18.934s including waiting). Image size: 1335613541 bytes.
10:55:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh	Started	Started container kserve-container
10:55:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb	Created	Created container: kserve-container
10:55:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb	Started	Started container kserve-container
10:55:07	kserve-ci-e2e-test	kubelet	success-200-isvc-6db9e-predictor-58b664446b-fmthr	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 5.058s (5.058s including waiting). Image size: 211946088 bytes.
10:55:07	kserve-ci-e2e-test	kubelet	error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g	Started	Started container kube-rbac-proxy
10:55:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 5.4s (5.4s including waiting). Image size: 211946088 bytes.
10:55:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb	Started	Started container kube-rbac-proxy
10:55:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb	Created	Created container: kube-rbac-proxy
10:55:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 5.111s (5.111s including waiting). Image size: 211946088 bytes.
10:55:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh	Created	Created container: kube-rbac-proxy
10:55:07	kserve-ci-e2e-test	kubelet	success-200-isvc-6db9e-predictor-58b664446b-fmthr	Created	Created container: kube-rbac-proxy
10:55:07	kserve-ci-e2e-test	kubelet	success-200-isvc-6db9e-predictor-58b664446b-fmthr	Started	Started container kube-rbac-proxy
10:55:07	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh	Started	Started container kube-rbac-proxy
10:55:07	kserve-ci-e2e-test	kubelet	error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 5.289s (5.289s including waiting). Image size: 211946088 bytes.
10:55:07	kserve-ci-e2e-test	kubelet	error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g	Created	Created container: kube-rbac-proxy
10:55:12 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-6db9e-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
10:55:12 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-6db9e-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
10:55:22	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-j57gn	Created	Created container: kserve-container
10:55:22	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-j57gn	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 31.346s (31.346s including waiting). Image size: 1306898171 bytes.
10:55:22	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-j57gn	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
10:55:22	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-j57gn	Started	Started container kserve-container
10:55:22	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-j57gn	Started	Started container kube-rbac-proxy
10:55:22	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-j57gn	Created	Created container: kube-rbac-proxy
10:55:28 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
10:55:28 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
10:55:28 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
10:55:28 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
10:55:43 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-6db9e-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
10:55:43 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-6db9e-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
10:55:54	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-6db9e	InferenceServiceReady	InferenceService [error-404-isvc-6db9e] is Ready
10:55:54 (x5)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-6db9e	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
10:55:58 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
10:55:58 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
10:56:04	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-6db9e	InferenceServiceReady	InferenceService [success-200-isvc-6db9e] is Ready
10:56:04 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-6db9e	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
10:56:13 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-6db9e-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
10:56:13 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-6db9e-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
10:56:20 (x8)	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-j57gn	Unhealthy	Readiness probe failed: dial tcp 10.133.0.24:8080: connect: connection refused
10:56:22	kserve-ci-e2e-test	kubelet	switch-graph-6db9e-5cb89b98dc-7x78t	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "switch-graph-6db9e-serving-cert" not found
10:56:22	kserve-ci-e2e-test	InferenceGraphController	switch-graph-6db9e	InferenceGraphReady	InferenceGraph [switch-graph-6db9e] is Ready
10:56:22	kserve-ci-e2e-test	replicaset-controller	switch-graph-6db9e-5cb89b98dc	SuccessfulCreate	Created pod: switch-graph-6db9e-5cb89b98dc-7x78t
10:56:22	kserve-ci-e2e-test	deployment-controller	switch-graph-6db9e	ScalingReplicaSet	Scaled up replica set switch-graph-6db9e-5cb89b98dc from 0 to 1
10:56:23	kserve-ci-e2e-test	multus	switch-graph-6db9e-5cb89b98dc-7x78t	AddedInterface	Add eth0 [10.134.0.26/23] from ovn-kubernetes
10:56:23	kserve-ci-e2e-test	kubelet	switch-graph-6db9e-5cb89b98dc-7x78t	Pulling	Pulling image "quay.io/opendatahub/kserve-router@sha256:0e1a169e211c4cdcd19b5d9ae2b04ed590c1bcb2f57424f5c6e7d0d0f2faa1f0"
10:56:26	kserve-ci-e2e-test	kubelet	switch-graph-6db9e-5cb89b98dc-7x78t	Started	Started container switch-graph-6db9e
10:56:26	kserve-ci-e2e-test	kubelet	switch-graph-6db9e-5cb89b98dc-7x78t	Created	Created container: switch-graph-6db9e
10:56:26	kserve-ci-e2e-test	kubelet	switch-graph-6db9e-5cb89b98dc-7x78t	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:0e1a169e211c4cdcd19b5d9ae2b04ed590c1bcb2f57424f5c6e7d0d0f2faa1f0" in 2.238s (2.238s including waiting). Image size: 218077281 bytes.
10:56:27 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-6db9e-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
10:56:27 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-6db9e-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
10:56:30 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-graph	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
10:56:30	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-graph	InferenceServiceReady	InferenceService [isvc-xgboost-graph] is Ready
10:56:32	kserve-ci-e2e-test	kubelet	switch-graph-6db9e-5cb89b98dc-7x78t	Killing	Stopping container switch-graph-6db9e
10:56:33 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
10:56:33 (x8)	kserve-ci-e2e-test	kubelet	success-200-isvc-6db9e-predictor-58b664446b-fmthr	Unhealthy	Readiness probe failed: dial tcp 10.133.0.21:8080: connect: connection refused
10:56:33	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-ff4f8	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-ff4f8": the object has been modified; please apply your changes to the latest version and try again
10:56:33	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-ff4f8	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-ff4f8": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-ff4f8": the object has been modified; please apply your changes to the latest version and try again
10:56:33	kserve-ci-e2e-test	deployment-controller	error-404-isvc-ff4f8-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-ff4f8-predictor-557dbbf4ff from 0 to 1
10:56:33	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-ff4f8-predictor-557dbbf4ff	SuccessfulCreate	Created pod: error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg
10:56:33	kserve-ci-e2e-test	kubelet	success-200-isvc-6db9e-predictor-58b664446b-fmthr	Unhealthy	Readiness probe failed: Get "https://10.133.0.21:8643/healthz": dial tcp 10.133.0.21:8643: connect: connection refused
10:56:33	kserve-ci-e2e-test	kubelet	error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g	Unhealthy	Readiness probe failed: Get "https://10.133.0.23:8643/healthz": dial tcp 10.133.0.23:8643: connect: connection refused
10:56:33	kserve-ci-e2e-test	kubelet	error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg	Started	Started container kube-rbac-proxy
10:56:33	kserve-ci-e2e-test	kubelet	error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg	Created	Created container: kube-rbac-proxy
10:56:33	kserve-ci-e2e-test	kubelet	error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
10:56:33	kserve-ci-e2e-test	kubelet	error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg	Started	Started container kserve-container
10:56:33	kserve-ci-e2e-test	kubelet	error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg	Created	Created container: kserve-container
10:56:33	kserve-ci-e2e-test	kubelet	error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1738" already present on machine
10:56:33	kserve-ci-e2e-test	multus	error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg	AddedInterface	Add eth0 [10.133.0.27/23] from ovn-kubernetes
10:56:33	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-ff4f8	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-ff4f8": the object has been modified; please apply your changes to the latest version and try again
10:56:33	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-ff4f8	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-ff4f8": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-ff4f8": the object has been modified; please apply your changes to the latest version and try again
10:56:33	kserve-ci-e2e-test	kubelet	error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g	Killing	Stopping container kube-rbac-proxy
10:56:33	kserve-ci-e2e-test	kubelet	error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g	Killing	Stopping container kserve-container
10:56:33	kserve-ci-e2e-test	deployment-controller	success-200-isvc-ff4f8-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-ff4f8-predictor-67c88c6d66 from 0 to 1
10:56:33	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-2	InferenceServiceReady	InferenceService [isvc-sklearn-graph-2] is Ready
10:56:33	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-ff4f8-predictor-67c88c6d66	SuccessfulCreate	Created pod: success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9
10:56:33	kserve-ci-e2e-test	kubelet	success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9	Started	Started container kube-rbac-proxy
10:56:33	kserve-ci-e2e-test	kubelet	success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9	Created	Created container: kube-rbac-proxy
10:56:33	kserve-ci-e2e-test	kubelet	success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
10:56:33	kserve-ci-e2e-test	kubelet	success-200-isvc-6db9e-predictor-58b664446b-fmthr	Killing	Stopping container kube-rbac-proxy
10:56:33	kserve-ci-e2e-test	kubelet	success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9	Started	Started container kserve-container
10:56:33	kserve-ci-e2e-test	kubelet	success-200-isvc-6db9e-predictor-58b664446b-fmthr	Killing	Stopping container kserve-container
10:56:33	kserve-ci-e2e-test	kubelet	success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9	Created	Created container: kserve-container
10:56:33	kserve-ci-e2e-test	kubelet	success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1738" already present on machine
10:56:33	kserve-ci-e2e-test	multus	success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9	AddedInterface	Add eth0 [10.133.0.26/23] from ovn-kubernetes
10:56:33 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-6db9e-predictor-6b44c644f9-7tc4g	Unhealthy	Readiness probe failed: dial tcp 10.133.0.23:8080: connect: connection refused
10:56:35	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-1	InferenceServiceReady	InferenceService [isvc-sklearn-graph-1] is Ready
10:56:35 (x7)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
10:56:48	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-ff4f8-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
10:56:48	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-ff4f8-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
10:56:58 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
10:56:58 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
10:56:58 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
10:56:58 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
10:56:58 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
10:56:58 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
10:57:02	kserve-ci-e2e-test	replicaset-controller	model-chainer-7fdbd8cf64	SuccessfulCreate	Created pod: model-chainer-7fdbd8cf64-8vlgf
10:57:02 (x6)	kserve-ci-e2e-test	kubelet	switch-graph-6db9e-5cb89b98dc-7x78t	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
10:57:02	kserve-ci-e2e-test	deployment-controller	model-chainer	ScalingReplicaSet	Scaled up replica set model-chainer-7fdbd8cf64 from 0 to 1
10:57:02	kserve-ci-e2e-test	InferenceGraphController	model-chainer	InferenceGraphReady	InferenceGraph [model-chainer] is Ready
10:57:03	kserve-ci-e2e-test	kubelet	model-chainer-7fdbd8cf64-8vlgf	Created	Created container: model-chainer
10:57:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-ff4f8-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
10:57:03	kserve-ci-e2e-test	multus	model-chainer-7fdbd8cf64-8vlgf	AddedInterface	Add eth0 [10.134.0.27/23] from ovn-kubernetes
10:57:03	kserve-ci-e2e-test	kubelet	model-chainer-7fdbd8cf64-8vlgf	Started	Started container model-chainer
10:57:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-ff4f8-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
10:57:03	kserve-ci-e2e-test	kubelet	model-chainer-7fdbd8cf64-8vlgf	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:0e1a169e211c4cdcd19b5d9ae2b04ed590c1bcb2f57424f5c6e7d0d0f2faa1f0" already present on machine
10:57:11 (x6)	kserve-ci-e2e-test	kubelet	success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9	Unhealthy	Readiness probe failed: dial tcp 10.133.0.26:8080: connect: connection refused
10:57:12	kserve-ci-e2e-test	kubelet	model-chainer-7fdbd8cf64-8vlgf	Killing	Stopping container model-chainer
10:57:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-j57gn	Killing	Stopping container kserve-container
10:57:13	kserve-ci-e2e-test	multus	error-404-isvc-074d2-predictor-576646885c-d8cfp	AddedInterface	Add eth0 [10.133.0.29/23] from ovn-kubernetes
10:57:13	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-074d2-predictor-6f6bcf5df6	SuccessfulCreate	Created pod: success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp
10:57:13	kserve-ci-e2e-test	kubelet	success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp	Started	Started container kube-rbac-proxy
10:57:13	kserve-ci-e2e-test	kubelet	error-404-isvc-074d2-predictor-576646885c-d8cfp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
10:57:13	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-j57gn	Killing	Stopping container kube-rbac-proxy
10:57:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb	Killing	Stopping container kube-rbac-proxy
10:57:13	kserve-ci-e2e-test	kubelet	error-404-isvc-074d2-predictor-576646885c-d8cfp	Started	Started container kserve-container
10:57:13	kserve-ci-e2e-test	kubelet	success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp	Created	Created container: kube-rbac-proxy
10:57:13	kserve-ci-e2e-test	kubelet	error-404-isvc-074d2-predictor-576646885c-d8cfp	Created	Created container: kserve-container
10:57:13	kserve-ci-e2e-test	deployment-controller	success-200-isvc-074d2-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-074d2-predictor-6f6bcf5df6 from 0 to 1
10:57:13	kserve-ci-e2e-test	kubelet	success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
10:57:13	kserve-ci-e2e-test	kubelet	error-404-isvc-074d2-predictor-576646885c-d8cfp	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1738" already present on machine
10:57:13	kserve-ci-e2e-test	kubelet	error-404-isvc-074d2-predictor-576646885c-d8cfp	Created	Created container: kube-rbac-proxy
10:57:13	kserve-ci-e2e-test	kubelet	success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp	Started	Started container kserve-container
10:57:13	kserve-ci-e2e-test	kubelet	success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp	Created	Created container: kserve-container
10:57:13	kserve-ci-e2e-test	kubelet	error-404-isvc-074d2-predictor-576646885c-d8cfp	Started	Started container kube-rbac-proxy
10:57:13	kserve-ci-e2e-test	kubelet	success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1738" already present on machine
10:57:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh	Killing	Stopping container kube-rbac-proxy
10:57:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh	Killing	Stopping container kserve-container
10:57:13 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh	Unhealthy	Readiness probe failed: dial tcp 10.133.0.25:8080: connect: connection refused
10:57:13	kserve-ci-e2e-test	multus	success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp	AddedInterface	Add eth0 [10.133.0.28/23] from ovn-kubernetes
10:57:13	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb	Killing	Stopping container kserve-container
10:57:13	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-074d2-predictor-576646885c	SuccessfulCreate	Created pod: error-404-isvc-074d2-predictor-576646885c-d8cfp
10:57:13	kserve-ci-e2e-test	deployment-controller	error-404-isvc-074d2-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-074d2-predictor-576646885c from 0 to 1
10:57:13	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-074d2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-074d2": the object has been modified; please apply your changes to the latest version and try again
10:57:13	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-074d2	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-074d2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-074d2": the object has been modified; please apply your changes to the latest version and try again
10:57:14 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb	Unhealthy	Readiness probe failed: dial tcp 10.133.0.22:8080: connect: connection refused
10:57:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-5cb7bb944-h7qvb	Unhealthy	Readiness probe failed: Get "https://10.133.0.22:8643/healthz": dial tcp 10.133.0.22:8643: connect: connection refused
10:57:15	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-j57gn	Unhealthy	Readiness probe failed: Get "https://10.133.0.24:8643/healthz": dial tcp 10.133.0.24:8643: connect: connection refused
10:57:18 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-d8594dbb4-w5wlh	Unhealthy	Readiness probe failed: Get "https://10.133.0.25:8643/healthz": dial tcp 10.133.0.25:8643: connect: connection refused
10:57:20	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-ff4f8	InferenceServiceReady	InferenceService [error-404-isvc-ff4f8] is Ready
10:57:20 (x5)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-ff4f8	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
10:57:21 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-ff4f8	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
10:57:21	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-ff4f8	InferenceServiceReady	InferenceService [success-200-isvc-ff4f8] is Ready
10:57:33	kserve-ci-e2e-test	kubelet	switch-graph-ff4f8-fc4884fc4-c9572	Created	Created container: switch-graph-ff4f8
10:57:33	kserve-ci-e2e-test	kubelet	switch-graph-ff4f8-fc4884fc4-c9572	Started	Started container switch-graph-ff4f8
10:57:33	kserve-ci-e2e-test	multus	switch-graph-ff4f8-fc4884fc4-c9572	AddedInterface	Add eth0 [10.134.0.28/23] from ovn-kubernetes
10:57:33	kserve-ci-e2e-test	InferenceGraphController	switch-graph-ff4f8	InferenceGraphReady	InferenceGraph [switch-graph-ff4f8] is Ready
10:57:33	kserve-ci-e2e-test	deployment-controller	switch-graph-ff4f8	ScalingReplicaSet	Scaled up replica set switch-graph-ff4f8-fc4884fc4 from 0 to 1
10:57:33	kserve-ci-e2e-test	replicaset-controller	switch-graph-ff4f8-fc4884fc4	SuccessfulCreate	Created pod: switch-graph-ff4f8-fc4884fc4-c9572
10:57:33	kserve-ci-e2e-test	kubelet	switch-graph-ff4f8-fc4884fc4-c9572	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:0e1a169e211c4cdcd19b5d9ae2b04ed590c1bcb2f57424f5c6e7d0d0f2faa1f0" already present on machine
10:57:33 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-ff4f8-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
10:57:33 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-ff4f8-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
10:57:39 (x6)	kserve-ci-e2e-test	kubelet	model-chainer-7fdbd8cf64-8vlgf	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
10:57:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-074d2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
10:57:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-074d2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
10:57:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-074d2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
10:57:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-074d2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
10:57:48 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-ff4f8-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
10:57:48 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-ff4f8-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
10:58:00 (x5)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-074d2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
10:58:00	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-074d2	InferenceServiceReady	InferenceService [error-404-isvc-074d2] is Ready
10:58:01 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-074d2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
10:58:01	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-074d2	InferenceServiceReady	InferenceService [success-200-isvc-074d2] is Ready
10:58:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-ff4f8	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
10:58:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-ff4f8	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
10:58:13	kserve-ci-e2e-test	kubelet	sequence-graph-074d2-574bc49b98-k4s78	Created	Created container: sequence-graph-074d2
10:58:13	kserve-ci-e2e-test	InferenceGraphController	sequence-graph-074d2	InferenceGraphReady	InferenceGraph [sequence-graph-074d2] is Ready
10:58:13	kserve-ci-e2e-test	multus	sequence-graph-074d2-574bc49b98-k4s78	AddedInterface	Add eth0 [10.134.0.29/23] from ovn-kubernetes
10:58:13	kserve-ci-e2e-test	kubelet	sequence-graph-074d2-574bc49b98-k4s78	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:0e1a169e211c4cdcd19b5d9ae2b04ed590c1bcb2f57424f5c6e7d0d0f2faa1f0" already present on machine
10:58:13	kserve-ci-e2e-test	deployment-controller	sequence-graph-074d2	ScalingReplicaSet	Scaled up replica set sequence-graph-074d2-574bc49b98 from 0 to 1
10:58:13	kserve-ci-e2e-test	kubelet	sequence-graph-074d2-574bc49b98-k4s78	Started	Started container sequence-graph-074d2
10:58:13	kserve-ci-e2e-test	replicaset-controller	sequence-graph-074d2-574bc49b98	SuccessfulCreate	Created pod: sequence-graph-074d2-574bc49b98-k4s78
10:58:28 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-074d2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
10:58:28 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-074d2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
10:58:43 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-074d2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
10:58:43 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-074d2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
10:58:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-074d2	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
10:58:43 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-074d2	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:05:47	kserve-ci-e2e-test	kubelet	switch-graph-ff4f8-fc4884fc4-c9572	Killing	Stopping container switch-graph-ff4f8
11:05:47	kserve-ci-e2e-test	kubelet	success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9	Killing	Stopping container kube-rbac-proxy
11:05:47	kserve-ci-e2e-test	kubelet	success-200-isvc-ff4f8-predictor-67c88c6d66-jxbd9	Killing	Stopping container kserve-container
11:05:47	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-cf436	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-cf436": the object has been modified; please apply your changes to the latest version and try again
11:05:47	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-cf436	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-cf436": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-cf436": the object has been modified; please apply your changes to the latest version and try again
11:05:47	kserve-ci-e2e-test	deployment-controller	success-200-isvc-cf436-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-cf436-predictor-69cbcd7974 from 0 to 1
11:05:47	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-cf436-predictor-69cbcd7974	SuccessfulCreate	Created pod: success-200-isvc-cf436-predictor-69cbcd7974-z8klj
11:05:48	kserve-ci-e2e-test	kubelet	error-404-isvc-cf436-predictor-777f9bff5f-j9df6	Started	Started container kube-rbac-proxy
11:05:48	kserve-ci-e2e-test	deployment-controller	error-404-isvc-cf436-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-cf436-predictor-777f9bff5f from 0 to 1
11:05:48	kserve-ci-e2e-test	kubelet	error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg	Killing	Stopping container kube-rbac-proxy
11:05:48	kserve-ci-e2e-test	kubelet	error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg	Killing	Stopping container kserve-container
11:05:48	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-cf436-predictor-777f9bff5f	SuccessfulCreate	Created pod: error-404-isvc-cf436-predictor-777f9bff5f-j9df6
11:05:48	kserve-ci-e2e-test	multus	success-200-isvc-cf436-predictor-69cbcd7974-z8klj	AddedInterface	Add eth0 [10.133.0.30/23] from ovn-kubernetes
11:05:48	kserve-ci-e2e-test	kubelet	success-200-isvc-cf436-predictor-69cbcd7974-z8klj	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1738" already present on machine
11:05:48	kserve-ci-e2e-test	kubelet	success-200-isvc-cf436-predictor-69cbcd7974-z8klj	Created	Created container: kserve-container
11:05:48	kserve-ci-e2e-test	kubelet	success-200-isvc-cf436-predictor-69cbcd7974-z8klj	Started	Started container kserve-container
11:05:48	kserve-ci-e2e-test	kubelet	success-200-isvc-cf436-predictor-69cbcd7974-z8klj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
11:05:48	kserve-ci-e2e-test	kubelet	success-200-isvc-cf436-predictor-69cbcd7974-z8klj	Created	Created container: kube-rbac-proxy
11:05:48	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-cf436	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-cf436": the object has been modified; please apply your changes to the latest version and try again
11:05:48	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-cf436	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-cf436": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-cf436": the object has been modified; please apply your changes to the latest version and try again
11:05:48	kserve-ci-e2e-test	kubelet	error-404-isvc-cf436-predictor-777f9bff5f-j9df6	Created	Created container: kube-rbac-proxy
11:05:48	kserve-ci-e2e-test	kubelet	success-200-isvc-cf436-predictor-69cbcd7974-z8klj	Started	Started container kube-rbac-proxy
11:05:48	kserve-ci-e2e-test	kubelet	error-404-isvc-cf436-predictor-777f9bff5f-j9df6	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
11:05:48	kserve-ci-e2e-test	multus	error-404-isvc-cf436-predictor-777f9bff5f-j9df6	AddedInterface	Add eth0 [10.133.0.31/23] from ovn-kubernetes
11:05:48	kserve-ci-e2e-test	kubelet	error-404-isvc-cf436-predictor-777f9bff5f-j9df6	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1738" already present on machine
11:05:48	kserve-ci-e2e-test	kubelet	error-404-isvc-cf436-predictor-777f9bff5f-j9df6	Created	Created container: kserve-container
11:05:48	kserve-ci-e2e-test	kubelet	error-404-isvc-cf436-predictor-777f9bff5f-j9df6	Started	Started container kserve-container
11:05:50	kserve-ci-e2e-test	kubelet	error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg	Unhealthy	Readiness probe failed: Get "https://10.133.0.27:8643/healthz": dial tcp 10.133.0.27:8643: connect: connection refused
11:05:50 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-ff4f8-predictor-557dbbf4ff-fb8lg	Unhealthy	Readiness probe failed: dial tcp 10.133.0.27:8080: connect: connection refused
11:06:02	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-cf436-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:06:02	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-cf436-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:06:14 (x6)	kserve-ci-e2e-test	kubelet	switch-graph-ff4f8-fc4884fc4-c9572	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
11:06:18 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-cf436-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:06:18 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-cf436-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:06:25 (x6)	kserve-ci-e2e-test	kubelet	error-404-isvc-cf436-predictor-777f9bff5f-j9df6	Unhealthy	Readiness probe failed: dial tcp 10.133.0.31:8080: connect: connection refused
11:06:27	kserve-ci-e2e-test	kubelet	sequence-graph-074d2-574bc49b98-k4s78	Killing	Stopping container sequence-graph-074d2
11:06:28	kserve-ci-e2e-test	kubelet	error-404-isvc-30120-predictor-6c4f57456c-6h8b6	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1738" already present on machine
11:06:28	kserve-ci-e2e-test	kubelet	success-200-isvc-30120-predictor-74997b5b5d-6hgnc	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "success-200-isvc-30120-predictor-serving-cert" not found
11:06:28	kserve-ci-e2e-test	kubelet	error-404-isvc-30120-predictor-6c4f57456c-6h8b6	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
11:06:28	kserve-ci-e2e-test	kubelet	error-404-isvc-30120-predictor-6c4f57456c-6h8b6	Started	Started container kserve-container
11:06:28	kserve-ci-e2e-test	kubelet	error-404-isvc-30120-predictor-6c4f57456c-6h8b6	Created	Created container: kserve-container
11:06:28	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-30120	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-30120": the object has been modified; please apply your changes to the latest version and try again
11:06:28	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-30120	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-30120": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-30120": the object has been modified; please apply your changes to the latest version and try again
11:06:28	kserve-ci-e2e-test	kubelet	error-404-isvc-30120-predictor-6c4f57456c-6h8b6	Started	Started container kube-rbac-proxy
11:06:28	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-30120-predictor-6c4f57456c	SuccessfulCreate	Created pod: error-404-isvc-30120-predictor-6c4f57456c-6h8b6
11:06:28	kserve-ci-e2e-test	deployment-controller	success-200-isvc-30120-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-30120-predictor-74997b5b5d from 0 to 1
11:06:28	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-30120-predictor-74997b5b5d	SuccessfulCreate	Created pod: success-200-isvc-30120-predictor-74997b5b5d-6hgnc
11:06:28	kserve-ci-e2e-test	deployment-controller	error-404-isvc-30120-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-30120-predictor-6c4f57456c from 0 to 1
11:06:28	kserve-ci-e2e-test	kubelet	error-404-isvc-30120-predictor-6c4f57456c-6h8b6	Created	Created container: kube-rbac-proxy
11:06:28	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-30120	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-30120": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-30120": the object has been modified; please apply your changes to the latest version and try again
11:06:28	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-30120	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-30120": the object has been modified; please apply your changes to the latest version and try again
11:06:28	kserve-ci-e2e-test	kubelet	error-404-isvc-074d2-predictor-576646885c-d8cfp	Killing	Stopping container kserve-container
11:06:28	kserve-ci-e2e-test	kubelet	error-404-isvc-074d2-predictor-576646885c-d8cfp	Killing	Stopping container kube-rbac-proxy
11:06:28	kserve-ci-e2e-test	kubelet	success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp	Killing	Stopping container kserve-container
11:06:28	kserve-ci-e2e-test	multus	error-404-isvc-30120-predictor-6c4f57456c-6h8b6	AddedInterface	Add eth0 [10.133.0.33/23] from ovn-kubernetes
11:06:28	kserve-ci-e2e-test	kubelet	success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp	Killing	Stopping container kube-rbac-proxy
11:06:29	kserve-ci-e2e-test	kubelet	success-200-isvc-30120-predictor-74997b5b5d-6hgnc	Created	Created container: kube-rbac-proxy
11:06:29	kserve-ci-e2e-test	kubelet	success-200-isvc-30120-predictor-74997b5b5d-6hgnc	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1738" already present on machine
11:06:29	kserve-ci-e2e-test	kubelet	success-200-isvc-30120-predictor-74997b5b5d-6hgnc	Started	Started container kube-rbac-proxy
11:06:29	kserve-ci-e2e-test	multus	success-200-isvc-30120-predictor-74997b5b5d-6hgnc	AddedInterface	Add eth0 [10.133.0.32/23] from ovn-kubernetes
11:06:29	kserve-ci-e2e-test	kubelet	success-200-isvc-30120-predictor-74997b5b5d-6hgnc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
11:06:29	kserve-ci-e2e-test	kubelet	success-200-isvc-30120-predictor-74997b5b5d-6hgnc	Created	Created container: kserve-container
11:06:29	kserve-ci-e2e-test	kubelet	success-200-isvc-30120-predictor-74997b5b5d-6hgnc	Started	Started container kserve-container
11:06:30	kserve-ci-e2e-test	kubelet	error-404-isvc-074d2-predictor-576646885c-d8cfp	Unhealthy	Readiness probe failed: Get "https://10.133.0.29:8643/healthz": dial tcp 10.133.0.29:8643: connect: connection refused
11:06:30 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-074d2-predictor-576646885c-d8cfp	Unhealthy	Readiness probe failed: dial tcp 10.133.0.29:8080: connect: connection refused
11:06:31	kserve-ci-e2e-test	kubelet	success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp	Unhealthy	Readiness probe failed: Get "https://10.133.0.28:8643/healthz": dial tcp 10.133.0.28:8643: connect: connection refused
11:06:31 (x7)	kserve-ci-e2e-test	kubelet	success-200-isvc-074d2-predictor-6f6bcf5df6-7bnmp	Unhealthy	Readiness probe failed: dial tcp 10.133.0.28:8080: connect: connection refused
11:06:35 (x5)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-cf436	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
11:06:35 (x7)	kserve-ci-e2e-test	kubelet	success-200-isvc-cf436-predictor-69cbcd7974-z8klj	Unhealthy	Readiness probe failed: dial tcp 10.133.0.30:8080: connect: connection refused
11:06:35	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-cf436	InferenceServiceReady	InferenceService [error-404-isvc-cf436] is Ready
11:06:45 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-cf436	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
11:06:45	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-cf436	InferenceServiceReady	InferenceService [success-200-isvc-cf436] is Ready
11:06:55 (x6)	kserve-ci-e2e-test	kubelet	sequence-graph-074d2-574bc49b98-k4s78	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
11:06:58 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-30120-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:06:58 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-30120-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:06:58 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-30120-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:06:58	kserve-ci-e2e-test	kubelet	ensemble-graph-cf436-7d4dbbfd97-vqcsn	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "ensemble-graph-cf436-serving-cert" not found
11:06:58 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-30120-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:06:58	kserve-ci-e2e-test	InferenceGraphController	ensemble-graph-cf436	InferenceGraphReady	InferenceGraph [ensemble-graph-cf436] is Ready
11:06:58	kserve-ci-e2e-test	deployment-controller	ensemble-graph-cf436	ScalingReplicaSet	Scaled up replica set ensemble-graph-cf436-7d4dbbfd97 from 0 to 1
11:06:58	kserve-ci-e2e-test	replicaset-controller	ensemble-graph-cf436-7d4dbbfd97	SuccessfulCreate	Created pod: ensemble-graph-cf436-7d4dbbfd97-vqcsn
11:06:59	kserve-ci-e2e-test	kubelet	ensemble-graph-cf436-7d4dbbfd97-vqcsn	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:0e1a169e211c4cdcd19b5d9ae2b04ed590c1bcb2f57424f5c6e7d0d0f2faa1f0" already present on machine
11:06:59	kserve-ci-e2e-test	kubelet	ensemble-graph-cf436-7d4dbbfd97-vqcsn	Created	Created container: ensemble-graph-cf436
11:06:59	kserve-ci-e2e-test	kubelet	ensemble-graph-cf436-7d4dbbfd97-vqcsn	Started	Started container ensemble-graph-cf436
11:06:59	kserve-ci-e2e-test	multus	ensemble-graph-cf436-7d4dbbfd97-vqcsn	AddedInterface	Add eth0 [10.134.0.30/23] from ovn-kubernetes
11:07:02 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-cf436-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:07:02 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-cf436-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:07:03 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-cf436-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:07:03 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-cf436-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:07:06 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-30120-predictor-6c4f57456c-6h8b6	Unhealthy	Readiness probe failed: dial tcp 10.133.0.33:8080: connect: connection refused
11:07:08	kserve-ci-e2e-test	kubelet	success-200-isvc-cf436-predictor-69cbcd7974-z8klj	Killing	Stopping container kserve-container
11:07:08	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-d6666-predictor-8674b65f5d	SuccessfulCreate	Created pod: success-200-isvc-d6666-predictor-8674b65f5d-8hxmp
11:07:08	kserve-ci-e2e-test	kubelet	ensemble-graph-cf436-7d4dbbfd97-vqcsn	Killing	Stopping container ensemble-graph-cf436
11:07:08	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-d6666	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-d6666": the object has been modified; please apply your changes to the latest version and try again
11:07:08	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-d6666	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-d6666": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-d6666": the object has been modified; please apply your changes to the latest version and try again
11:07:08	kserve-ci-e2e-test	kubelet	error-404-isvc-cf436-predictor-777f9bff5f-j9df6	Killing	Stopping container kube-rbac-proxy
11:07:08	kserve-ci-e2e-test	kubelet	success-200-isvc-cf436-predictor-69cbcd7974-z8klj	Killing	Stopping container kube-rbac-proxy
11:07:08	kserve-ci-e2e-test	deployment-controller	error-404-isvc-d6666-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-d6666-predictor-774b85585f from 0 to 1
11:07:08	kserve-ci-e2e-test	kubelet	error-404-isvc-cf436-predictor-777f9bff5f-j9df6	Killing	Stopping container kserve-container
11:07:08	kserve-ci-e2e-test	kubelet	error-404-isvc-d6666-predictor-774b85585f-t7hr4	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "error-404-isvc-d6666-predictor-serving-cert" not found
11:07:08	kserve-ci-e2e-test	multus	success-200-isvc-d6666-predictor-8674b65f5d-8hxmp	AddedInterface	Add eth0 [10.133.0.34/23] from ovn-kubernetes
11:07:08	kserve-ci-e2e-test	kubelet	success-200-isvc-d6666-predictor-8674b65f5d-8hxmp	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1738" already present on machine
11:07:08	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-d6666-predictor-774b85585f	SuccessfulCreate	Created pod: error-404-isvc-d6666-predictor-774b85585f-t7hr4
11:07:08	kserve-ci-e2e-test	kubelet	success-200-isvc-d6666-predictor-8674b65f5d-8hxmp	Created	Created container: kserve-container
11:07:08	kserve-ci-e2e-test	kubelet	success-200-isvc-d6666-predictor-8674b65f5d-8hxmp	Started	Started container kserve-container
11:07:08	kserve-ci-e2e-test	kubelet	success-200-isvc-d6666-predictor-8674b65f5d-8hxmp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
11:07:08	kserve-ci-e2e-test	kubelet	success-200-isvc-d6666-predictor-8674b65f5d-8hxmp	Created	Created container: kube-rbac-proxy
11:07:08	kserve-ci-e2e-test	kubelet	success-200-isvc-d6666-predictor-8674b65f5d-8hxmp	Started	Started container kube-rbac-proxy
11:07:08	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-d6666	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-d6666": the object has been modified; please apply your changes to the latest version and try again
11:07:08	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-d6666	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-d6666": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-d6666": the object has been modified; please apply your changes to the latest version and try again
11:07:08	kserve-ci-e2e-test	deployment-controller	success-200-isvc-d6666-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-d6666-predictor-8674b65f5d from 0 to 1
11:07:09	kserve-ci-e2e-test	kubelet	error-404-isvc-d6666-predictor-774b85585f-t7hr4	Started	Started container kube-rbac-proxy
11:07:09	kserve-ci-e2e-test	kubelet	error-404-isvc-d6666-predictor-774b85585f-t7hr4	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1738" already present on machine
11:07:09	kserve-ci-e2e-test	kubelet	error-404-isvc-d6666-predictor-774b85585f-t7hr4	Created	Created container: kube-rbac-proxy
11:07:09	kserve-ci-e2e-test	multus	error-404-isvc-d6666-predictor-774b85585f-t7hr4	AddedInterface	Add eth0 [10.133.0.35/23] from ovn-kubernetes
11:07:09	kserve-ci-e2e-test	kubelet	error-404-isvc-d6666-predictor-774b85585f-t7hr4	Created	Created container: kserve-container
11:07:09	kserve-ci-e2e-test	kubelet	error-404-isvc-d6666-predictor-774b85585f-t7hr4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
11:07:09	kserve-ci-e2e-test	kubelet	error-404-isvc-d6666-predictor-774b85585f-t7hr4	Started	Started container kserve-container
11:07:10	kserve-ci-e2e-test	kubelet	error-404-isvc-cf436-predictor-777f9bff5f-j9df6	Unhealthy	Readiness probe failed: Get "https://10.133.0.31:8643/healthz": dial tcp 10.133.0.31:8643: connect: connection refused
11:07:10	kserve-ci-e2e-test	kubelet	success-200-isvc-cf436-predictor-69cbcd7974-z8klj	Unhealthy	Readiness probe failed: Get "https://10.133.0.30:8643/healthz": dial tcp 10.133.0.30:8643: connect: connection refused
11:07:16 (x5)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-30120	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
11:07:16 (x7)	kserve-ci-e2e-test	kubelet	success-200-isvc-30120-predictor-74997b5b5d-6hgnc	Unhealthy	Readiness probe failed: dial tcp 10.133.0.32:8080: connect: connection refused
11:07:16	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-30120	InferenceServiceReady	InferenceService [error-404-isvc-30120] is Ready
11:07:26 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-30120	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
11:07:26	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-30120	InferenceServiceReady	InferenceService [success-200-isvc-30120] is Ready
11:07:35 (x6)	kserve-ci-e2e-test	kubelet	ensemble-graph-cf436-7d4dbbfd97-vqcsn	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
11:07:38	kserve-ci-e2e-test	replicaset-controller	sequence-graph-30120-7b65c6b7f	SuccessfulCreate	Created pod: sequence-graph-30120-7b65c6b7f-q2rbc
11:07:38	kserve-ci-e2e-test	deployment-controller	sequence-graph-30120	ScalingReplicaSet	Scaled up replica set sequence-graph-30120-7b65c6b7f from 0 to 1
11:07:38	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-d6666-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:07:38	kserve-ci-e2e-test	kubelet	sequence-graph-30120-7b65c6b7f-q2rbc	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "sequence-graph-30120-serving-cert" not found
11:07:38	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-d6666-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:07:38	kserve-ci-e2e-test	InferenceGraphController	sequence-graph-30120	InferenceGraphReady	InferenceGraph [sequence-graph-30120] is Ready
11:07:39	kserve-ci-e2e-test	multus	sequence-graph-30120-7b65c6b7f-q2rbc	AddedInterface	Add eth0 [10.134.0.31/23] from ovn-kubernetes
11:07:39	kserve-ci-e2e-test	kubelet	sequence-graph-30120-7b65c6b7f-q2rbc	Started	Started container sequence-graph-30120
11:07:39	kserve-ci-e2e-test	kubelet	sequence-graph-30120-7b65c6b7f-q2rbc	Created	Created container: sequence-graph-30120
11:07:39	kserve-ci-e2e-test	kubelet	sequence-graph-30120-7b65c6b7f-q2rbc	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:0e1a169e211c4cdcd19b5d9ae2b04ed590c1bcb2f57424f5c6e7d0d0f2faa1f0" already present on machine
11:07:43 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-30120-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:07:43 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-30120-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:07:43 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-30120-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:07:43 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-30120-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:07:46 (x6)	kserve-ci-e2e-test	kubelet	success-200-isvc-d6666-predictor-8674b65f5d-8hxmp	Unhealthy	Readiness probe failed: dial tcp 10.133.0.34:8080: connect: connection refused
11:07:48	kserve-ci-e2e-test	kubelet	error-404-isvc-70137-predictor-544d7497cf-k6qvv	Created	Created container: kserve-container
11:07:48	kserve-ci-e2e-test	multus	error-404-isvc-70137-predictor-544d7497cf-k6qvv	AddedInterface	Add eth0 [10.133.0.37/23] from ovn-kubernetes
11:07:48	kserve-ci-e2e-test	deployment-controller	error-404-isvc-70137-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-70137-predictor-544d7497cf from 0 to 1
11:07:48	kserve-ci-e2e-test	deployment-controller	success-200-isvc-70137-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-70137-predictor-77585cb866 from 0 to 1
11:07:48	kserve-ci-e2e-test	kubelet	error-404-isvc-70137-predictor-544d7497cf-k6qvv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
11:07:48	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-70137-predictor-77585cb866	SuccessfulCreate	Created pod: success-200-isvc-70137-predictor-77585cb866-kcdrt
11:07:48	kserve-ci-e2e-test	kubelet	error-404-isvc-30120-predictor-6c4f57456c-6h8b6	Killing	Stopping container kserve-container
11:07:48	kserve-ci-e2e-test	kubelet	success-200-isvc-70137-predictor-77585cb866-kcdrt	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "success-200-isvc-70137-predictor-serving-cert" not found
11:07:48	kserve-ci-e2e-test	kubelet	error-404-isvc-30120-predictor-6c4f57456c-6h8b6	Killing	Stopping container kube-rbac-proxy
11:07:48	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-70137-predictor-544d7497cf	SuccessfulCreate	Created pod: error-404-isvc-70137-predictor-544d7497cf-k6qvv
11:07:48	kserve-ci-e2e-test	kubelet	sequence-graph-30120-7b65c6b7f-q2rbc	Killing	Stopping container sequence-graph-30120
11:07:48	kserve-ci-e2e-test	kubelet	success-200-isvc-30120-predictor-74997b5b5d-6hgnc	Killing	Stopping container kube-rbac-proxy
11:07:48	kserve-ci-e2e-test	kubelet	error-404-isvc-70137-predictor-544d7497cf-k6qvv	Started	Started container kserve-container
11:07:48	kserve-ci-e2e-test	kubelet	error-404-isvc-70137-predictor-544d7497cf-k6qvv	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1738" already present on machine
11:07:48	kserve-ci-e2e-test	kubelet	success-200-isvc-30120-predictor-74997b5b5d-6hgnc	Killing	Stopping container kserve-container
11:07:49	kserve-ci-e2e-test	kubelet	success-200-isvc-70137-predictor-77585cb866-kcdrt	Created	Created container: kube-rbac-proxy
11:07:49	kserve-ci-e2e-test	kubelet	error-404-isvc-70137-predictor-544d7497cf-k6qvv	Created	Created container: kube-rbac-proxy
11:07:49	kserve-ci-e2e-test	kubelet	success-200-isvc-70137-predictor-77585cb866-kcdrt	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
11:07:49	kserve-ci-e2e-test	kubelet	success-200-isvc-70137-predictor-77585cb866-kcdrt	Created	Created container: kserve-container
11:07:49	kserve-ci-e2e-test	kubelet	success-200-isvc-70137-predictor-77585cb866-kcdrt	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1738" already present on machine
11:07:49	kserve-ci-e2e-test	kubelet	error-404-isvc-70137-predictor-544d7497cf-k6qvv	Started	Started container kube-rbac-proxy
11:07:49	kserve-ci-e2e-test	multus	success-200-isvc-70137-predictor-77585cb866-kcdrt	AddedInterface	Add eth0 [10.133.0.36/23] from ovn-kubernetes
11:07:49	kserve-ci-e2e-test	kubelet	success-200-isvc-70137-predictor-77585cb866-kcdrt	Started	Started container kserve-container
11:07:49	kserve-ci-e2e-test	kubelet	success-200-isvc-70137-predictor-77585cb866-kcdrt	Started	Started container kube-rbac-proxy
11:07:51	kserve-ci-e2e-test	kubelet	success-200-isvc-30120-predictor-74997b5b5d-6hgnc	Unhealthy	Readiness probe failed: Get "https://10.133.0.32:8643/healthz": dial tcp 10.133.0.32:8643: connect: connection refused
11:07:55 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-d6666	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
11:07:55	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-d6666	InferenceServiceReady	InferenceService [error-404-isvc-d6666] is Ready
11:07:56	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-d6666	InferenceServiceReady	InferenceService [success-200-isvc-d6666] is Ready
11:07:56 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-d6666	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
11:08:03	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-70137-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:08:03	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-70137-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:08:08	kserve-ci-e2e-test	kubelet	ensemble-graph-d6666-6bd466db88-jkkg7	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:0e1a169e211c4cdcd19b5d9ae2b04ed590c1bcb2f57424f5c6e7d0d0f2faa1f0" already present on machine
11:08:08	kserve-ci-e2e-test	InferenceGraphController	ensemble-graph-d6666	InferenceGraphReady	InferenceGraph [ensemble-graph-d6666] is Ready
11:08:08	kserve-ci-e2e-test	deployment-controller	ensemble-graph-d6666	ScalingReplicaSet	Scaled up replica set ensemble-graph-d6666-6bd466db88 from 0 to 1
11:08:08	kserve-ci-e2e-test	multus	ensemble-graph-d6666-6bd466db88-jkkg7	AddedInterface	Add eth0 [10.134.0.32/23] from ovn-kubernetes
11:08:08	kserve-ci-e2e-test	replicaset-controller	ensemble-graph-d6666-6bd466db88	SuccessfulCreate	Created pod: ensemble-graph-d6666-6bd466db88-jkkg7
11:08:08	kserve-ci-e2e-test	kubelet	ensemble-graph-d6666-6bd466db88-jkkg7	Created	Created container: ensemble-graph-d6666
11:08:08	kserve-ci-e2e-test	kubelet	ensemble-graph-d6666-6bd466db88-jkkg7	Started	Started container ensemble-graph-d6666
11:08:15 (x6)	kserve-ci-e2e-test	kubelet	sequence-graph-30120-7b65c6b7f-q2rbc	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
11:08:18 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-70137-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:08:18 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-70137-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:08:23 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-d6666-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:08:23 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-d6666-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:08:37	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-70137	InferenceServiceReady	InferenceService [error-404-isvc-70137] is Ready
11:08:37	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-70137	InferenceServiceReady	InferenceService [success-200-isvc-70137] is Ready
11:08:37 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-70137	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
11:08:37 (x6)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-70137	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
11:08:38 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	ensemble-graph-d6666	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:08:38 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	ensemble-graph-d6666	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:08:38 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-d6666-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:08:38 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-d6666-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:08:48	kserve-ci-e2e-test	deployment-controller	sequence-graph-70137	ScalingReplicaSet	Scaled up replica set sequence-graph-70137-5dcc4d4f8 from 0 to 1
11:08:48	kserve-ci-e2e-test	kubelet	sequence-graph-70137-5dcc4d4f8-8llvh	Started	Started container sequence-graph-70137
11:08:48	kserve-ci-e2e-test	InferenceGraphController	sequence-graph-70137	InferenceGraphReady	InferenceGraph [sequence-graph-70137] is Ready
11:08:48	kserve-ci-e2e-test	replicaset-controller	sequence-graph-70137-5dcc4d4f8	SuccessfulCreate	Created pod: sequence-graph-70137-5dcc4d4f8-8llvh
11:08:48	kserve-ci-e2e-test	multus	sequence-graph-70137-5dcc4d4f8-8llvh	AddedInterface	Add eth0 [10.134.0.33/23] from ovn-kubernetes
11:08:48	kserve-ci-e2e-test	kubelet	sequence-graph-70137-5dcc4d4f8-8llvh	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:0e1a169e211c4cdcd19b5d9ae2b04ed590c1bcb2f57424f5c6e7d0d0f2faa1f0" already present on machine
11:08:48	kserve-ci-e2e-test	kubelet	sequence-graph-70137-5dcc4d4f8-8llvh	Created	Created container: sequence-graph-70137
11:09:03	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-70137	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:09:03	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-70137	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:09:18 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-70137-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:09:18 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-70137-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:09:18	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-70137	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:09:18	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-70137	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:09:18 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-70137-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:09:18 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-70137-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:16:22	kserve-ci-e2e-test	kubelet	ensemble-graph-d6666-6bd466db88-jkkg7	Killing	Stopping container ensemble-graph-d6666
11:16:23	kserve-ci-e2e-test	kubelet	success-200-isvc-cdb7a-predictor-54d956f49b-dzphd	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1738" already present on machine
11:16:23	kserve-ci-e2e-test	kubelet	error-404-isvc-cdb7a-predictor-66db674dcc-48hvs	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
11:16:23	kserve-ci-e2e-test	kubelet	success-200-isvc-cdb7a-predictor-54d956f49b-dzphd	Created	Created container: kserve-container
11:16:23	kserve-ci-e2e-test	kubelet	error-404-isvc-d6666-predictor-774b85585f-t7hr4	Killing	Stopping container kube-rbac-proxy
11:16:23	kserve-ci-e2e-test	kubelet	error-404-isvc-d6666-predictor-774b85585f-t7hr4	Killing	Stopping container kserve-container
11:16:23	kserve-ci-e2e-test	kubelet	success-200-isvc-cdb7a-predictor-54d956f49b-dzphd	Started	Started container kserve-container
11:16:23	kserve-ci-e2e-test	kubelet	success-200-isvc-cdb7a-predictor-54d956f49b-dzphd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
11:16:23	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-cdb7a	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-cdb7a": the object has been modified; please apply your changes to the latest version and try again
11:16:23	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-cdb7a	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-cdb7a": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-cdb7a": the object has been modified; please apply your changes to the latest version and try again
11:16:23	kserve-ci-e2e-test	kubelet	success-200-isvc-d6666-predictor-8674b65f5d-8hxmp	Killing	Stopping container kube-rbac-proxy
11:16:23	kserve-ci-e2e-test	kubelet	success-200-isvc-d6666-predictor-8674b65f5d-8hxmp	Killing	Stopping container kserve-container
11:16:23	kserve-ci-e2e-test	deployment-controller	error-404-isvc-cdb7a-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-cdb7a-predictor-66db674dcc from 0 to 1
11:16:23	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-cdb7a-predictor-66db674dcc	SuccessfulCreate	Created pod: error-404-isvc-cdb7a-predictor-66db674dcc-48hvs
11:16:23	kserve-ci-e2e-test	kubelet	error-404-isvc-cdb7a-predictor-66db674dcc-48hvs	Started	Started container kube-rbac-proxy
11:16:23	kserve-ci-e2e-test	kubelet	error-404-isvc-cdb7a-predictor-66db674dcc-48hvs	Created	Created container: kube-rbac-proxy
11:16:23	kserve-ci-e2e-test	multus	success-200-isvc-cdb7a-predictor-54d956f49b-dzphd	AddedInterface	Add eth0 [10.133.0.38/23] from ovn-kubernetes
11:16:23	kserve-ci-e2e-test	kubelet	error-404-isvc-cdb7a-predictor-66db674dcc-48hvs	Started	Started container kserve-container
11:16:23	kserve-ci-e2e-test	kubelet	error-404-isvc-cdb7a-predictor-66db674dcc-48hvs	Created	Created container: kserve-container
11:16:23	kserve-ci-e2e-test	kubelet	error-404-isvc-cdb7a-predictor-66db674dcc-48hvs	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1738" already present on machine
11:16:23	kserve-ci-e2e-test	multus	error-404-isvc-cdb7a-predictor-66db674dcc-48hvs	AddedInterface	Add eth0 [10.133.0.39/23] from ovn-kubernetes
11:16:23	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-cdb7a	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-cdb7a": the object has been modified; please apply your changes to the latest version and try again
11:16:23	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-cdb7a	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-cdb7a": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-cdb7a": the object has been modified; please apply your changes to the latest version and try again
11:16:23	kserve-ci-e2e-test	deployment-controller	success-200-isvc-cdb7a-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-cdb7a-predictor-54d956f49b from 0 to 1
11:16:23	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-cdb7a-predictor-54d956f49b	SuccessfulCreate	Created pod: success-200-isvc-cdb7a-predictor-54d956f49b-dzphd
11:16:23	kserve-ci-e2e-test	kubelet	success-200-isvc-cdb7a-predictor-54d956f49b-dzphd	Started	Started container kube-rbac-proxy
11:16:23	kserve-ci-e2e-test	kubelet	success-200-isvc-cdb7a-predictor-54d956f49b-dzphd	Created	Created container: kube-rbac-proxy
11:16:25 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-d6666-predictor-774b85585f-t7hr4	Unhealthy	Readiness probe failed: dial tcp 10.133.0.35:8080: connect: connection refused
11:16:25	kserve-ci-e2e-test	kubelet	error-404-isvc-d6666-predictor-774b85585f-t7hr4	Unhealthy	Readiness probe failed: Get "https://10.133.0.35:8643/healthz": dial tcp 10.133.0.35:8643: connect: connection refused
11:16:50 (x6)	kserve-ci-e2e-test	kubelet	ensemble-graph-d6666-6bd466db88-jkkg7	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
11:16:53	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-cdb7a-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:16:53	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-cdb7a-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:16:53	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-cdb7a-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:16:53	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-cdb7a-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:17:01 (x6)	kserve-ci-e2e-test	kubelet	success-200-isvc-cdb7a-predictor-54d956f49b-dzphd	Unhealthy	Readiness probe failed: dial tcp 10.133.0.38:8080: connect: connection refused
11:17:01 (x6)	kserve-ci-e2e-test	kubelet	error-404-isvc-cdb7a-predictor-66db674dcc-48hvs	Unhealthy	Readiness probe failed: dial tcp 10.133.0.39:8080: connect: connection refused
11:17:03	kserve-ci-e2e-test	kubelet	sequence-graph-70137-5dcc4d4f8-8llvh	Killing	Stopping container sequence-graph-70137
11:17:03	kserve-ci-e2e-test	kubelet	error-404-isvc-bc774-predictor-8547545bdf-ktmgb	Created	Created container: kserve-container
11:17:03	kserve-ci-e2e-test	deployment-controller	error-404-isvc-bc774-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-bc774-predictor-8547545bdf from 0 to 1
11:17:03	kserve-ci-e2e-test	kubelet	success-200-isvc-70137-predictor-77585cb866-kcdrt	Killing	Stopping container kserve-container
11:17:03	kserve-ci-e2e-test	multus	success-200-isvc-bc774-predictor-5b6b775cdd-29slw	AddedInterface	Add eth0 [10.133.0.40/23] from ovn-kubernetes
11:17:03	kserve-ci-e2e-test	kubelet	error-404-isvc-70137-predictor-544d7497cf-k6qvv	Killing	Stopping container kserve-container
11:17:03	kserve-ci-e2e-test	kubelet	success-200-isvc-bc774-predictor-5b6b775cdd-29slw	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1738" already present on machine
11:17:03	kserve-ci-e2e-test	kubelet	error-404-isvc-bc774-predictor-8547545bdf-ktmgb	Started	Started container kube-rbac-proxy
11:17:03	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-bc774-predictor-8547545bdf	SuccessfulCreate	Created pod: error-404-isvc-bc774-predictor-8547545bdf-ktmgb
11:17:03	kserve-ci-e2e-test	deployment-controller	success-200-isvc-bc774-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-bc774-predictor-5b6b775cdd from 0 to 1
11:17:03	kserve-ci-e2e-test	kubelet	error-404-isvc-bc774-predictor-8547545bdf-ktmgb	Created	Created container: kube-rbac-proxy
11:17:03	kserve-ci-e2e-test	kubelet	error-404-isvc-bc774-predictor-8547545bdf-ktmgb	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
11:17:03	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-bc774	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-bc774": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-bc774": the object has been modified; please apply your changes to the latest version and try again
11:17:03	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-bc774	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-bc774": the object has been modified; please apply your changes to the latest version and try again
11:17:03	kserve-ci-e2e-test	kubelet	success-200-isvc-bc774-predictor-5b6b775cdd-29slw	Created	Created container: kserve-container
11:17:03	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-bc774-predictor-5b6b775cdd	SuccessfulCreate	Created pod: success-200-isvc-bc774-predictor-5b6b775cdd-29slw
11:17:03	kserve-ci-e2e-test	kubelet	success-200-isvc-bc774-predictor-5b6b775cdd-29slw	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
11:17:03	kserve-ci-e2e-test	kubelet	success-200-isvc-bc774-predictor-5b6b775cdd-29slw	Started	Started container kserve-container
11:17:03	kserve-ci-e2e-test	kubelet	error-404-isvc-bc774-predictor-8547545bdf-ktmgb	Started	Started container kserve-container
11:17:03	kserve-ci-e2e-test	kubelet	error-404-isvc-70137-predictor-544d7497cf-k6qvv	Killing	Stopping container kube-rbac-proxy
11:17:03	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-bc774	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-bc774": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-bc774": the object has been modified; please apply your changes to the latest version and try again
11:17:03	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-bc774	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-bc774": the object has been modified; please apply your changes to the latest version and try again
11:17:03	kserve-ci-e2e-test	multus	error-404-isvc-bc774-predictor-8547545bdf-ktmgb	AddedInterface	Add eth0 [10.133.0.41/23] from ovn-kubernetes
11:17:03	kserve-ci-e2e-test	kubelet	success-200-isvc-70137-predictor-77585cb866-kcdrt	Killing	Stopping container kube-rbac-proxy
11:17:03	kserve-ci-e2e-test	kubelet	error-404-isvc-bc774-predictor-8547545bdf-ktmgb	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1738" already present on machine
11:17:03	kserve-ci-e2e-test	kubelet	success-200-isvc-bc774-predictor-5b6b775cdd-29slw	Started	Started container kube-rbac-proxy
11:17:03	kserve-ci-e2e-test	kubelet	success-200-isvc-bc774-predictor-5b6b775cdd-29slw	Created	Created container: kube-rbac-proxy
11:17:07	kserve-ci-e2e-test	kubelet	success-200-isvc-70137-predictor-77585cb866-kcdrt	Unhealthy	Readiness probe failed: Get "https://10.133.0.36:8643/healthz": dial tcp 10.133.0.36:8643: connect: connection refused
11:17:07 (x7)	kserve-ci-e2e-test	kubelet	success-200-isvc-70137-predictor-77585cb866-kcdrt	Unhealthy	Readiness probe failed: dial tcp 10.133.0.36:8080: connect: connection refused
11:17:07	kserve-ci-e2e-test	kubelet	error-404-isvc-70137-predictor-544d7497cf-k6qvv	Unhealthy	Readiness probe failed: Get "https://10.133.0.37:8643/healthz": dial tcp 10.133.0.37:8643: connect: connection refused
11:17:07 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-70137-predictor-544d7497cf-k6qvv	Unhealthy	Readiness probe failed: dial tcp 10.133.0.37:8080: connect: connection refused
11:17:11	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-cdb7a	InferenceServiceReady	InferenceService [success-200-isvc-cdb7a] is Ready
11:17:11 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-cdb7a	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
11:17:11	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-cdb7a	InferenceServiceReady	InferenceService [error-404-isvc-cdb7a] is Ready
11:17:11 (x5)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-cdb7a	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
11:17:18	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-bc774-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:17:18	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-bc774-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:17:23	kserve-ci-e2e-test	InferenceGraphController	splitter-graph-cdb7a	InferenceGraphReady	InferenceGraph [splitter-graph-cdb7a] is Ready
11:17:23 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-cdb7a-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:17:23	kserve-ci-e2e-test	deployment-controller	splitter-graph-cdb7a	ScalingReplicaSet	Scaled up replica set splitter-graph-cdb7a-ffd5b4495 from 0 to 1
11:17:23	kserve-ci-e2e-test	replicaset-controller	splitter-graph-cdb7a-ffd5b4495	SuccessfulCreate	Created pod: splitter-graph-cdb7a-ffd5b4495-dstlk
11:17:23	kserve-ci-e2e-test	kubelet	splitter-graph-cdb7a-ffd5b4495-dstlk	Started	Started container splitter-graph-cdb7a
11:17:23	kserve-ci-e2e-test	kubelet	splitter-graph-cdb7a-ffd5b4495-dstlk	Created	Created container: splitter-graph-cdb7a
11:17:23	kserve-ci-e2e-test	kubelet	splitter-graph-cdb7a-ffd5b4495-dstlk	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:0e1a169e211c4cdcd19b5d9ae2b04ed590c1bcb2f57424f5c6e7d0d0f2faa1f0" already present on machine
11:17:23 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-cdb7a-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:17:23 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-cdb7a-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:17:23 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-cdb7a-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:17:23	kserve-ci-e2e-test	multus	splitter-graph-cdb7a-ffd5b4495-dstlk	AddedInterface	Add eth0 [10.134.0.34/23] from ovn-kubernetes
11:17:30 (x6)	kserve-ci-e2e-test	kubelet	sequence-graph-70137-5dcc4d4f8-8llvh	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
11:17:33	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-d1ffd	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-d1ffd": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-d1ffd": the object has been modified; please apply your changes to the latest version and try again
11:17:33	kserve-ci-e2e-test	kubelet	success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf	Started	Started container kserve-container
11:17:33	kserve-ci-e2e-test	deployment-controller	error-404-isvc-d1ffd-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-d1ffd-predictor-bb8d46565 from 0 to 1
11:17:33	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-d1ffd-predictor-bb8d46565	SuccessfulCreate	Created pod: error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2
11:17:33	kserve-ci-e2e-test	kubelet	error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "error-404-isvc-d1ffd-predictor-serving-cert" not found
11:17:33	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-d1ffd-predictor-759c7cbb88	SuccessfulCreate	Created pod: success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf
11:17:33	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-d1ffd	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-d1ffd": the object has been modified; please apply your changes to the latest version and try again
11:17:33	kserve-ci-e2e-test	kubelet	success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
11:17:33	kserve-ci-e2e-test	kubelet	success-200-isvc-cdb7a-predictor-54d956f49b-dzphd	Killing	Stopping container kube-rbac-proxy
11:17:33	kserve-ci-e2e-test	deployment-controller	success-200-isvc-d1ffd-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-d1ffd-predictor-759c7cbb88 from 0 to 1
11:17:33	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-d1ffd	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-d1ffd": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-d1ffd": the object has been modified; please apply your changes to the latest version and try again
11:17:33	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-d1ffd	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-d1ffd": the object has been modified; please apply your changes to the latest version and try again
11:17:33	kserve-ci-e2e-test	kubelet	success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf	Created	Created container: kserve-container
11:17:33	kserve-ci-e2e-test	kubelet	success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1738" already present on machine
11:17:33	kserve-ci-e2e-test	kubelet	splitter-graph-cdb7a-ffd5b4495-dstlk	Killing	Stopping container splitter-graph-cdb7a
11:17:33	kserve-ci-e2e-test	kubelet	success-200-isvc-cdb7a-predictor-54d956f49b-dzphd	Killing	Stopping container kserve-container
11:17:33	kserve-ci-e2e-test	multus	success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf	AddedInterface	Add eth0 [10.133.0.42/23] from ovn-kubernetes
11:17:33	kserve-ci-e2e-test	kubelet	error-404-isvc-cdb7a-predictor-66db674dcc-48hvs	Killing	Stopping container kserve-container
11:17:33	kserve-ci-e2e-test	kubelet	error-404-isvc-cdb7a-predictor-66db674dcc-48hvs	Killing	Stopping container kube-rbac-proxy
11:17:34	kserve-ci-e2e-test	multus	error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2	AddedInterface	Add eth0 [10.133.0.43/23] from ovn-kubernetes
11:17:34	kserve-ci-e2e-test	kubelet	success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf	Created	Created container: kube-rbac-proxy
11:17:34	kserve-ci-e2e-test	kubelet	error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2	Started	Started container kube-rbac-proxy
11:17:34	kserve-ci-e2e-test	kubelet	error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2	Created	Created container: kube-rbac-proxy
11:17:34	kserve-ci-e2e-test	kubelet	error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
11:17:34	kserve-ci-e2e-test	kubelet	success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf	Started	Started container kube-rbac-proxy
11:17:34	kserve-ci-e2e-test	kubelet	error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2	Started	Started container kserve-container
11:17:34	kserve-ci-e2e-test	kubelet	error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1738" already present on machine
11:17:34	kserve-ci-e2e-test	kubelet	error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2	Created	Created container: kserve-container
11:17:36	kserve-ci-e2e-test	kubelet	error-404-isvc-cdb7a-predictor-66db674dcc-48hvs	Unhealthy	Readiness probe failed: Get "https://10.133.0.39:8643/healthz": dial tcp 10.133.0.39:8643: connect: connection refused
11:17:36	kserve-ci-e2e-test	kubelet	success-200-isvc-cdb7a-predictor-54d956f49b-dzphd	Unhealthy	Readiness probe failed: Get "https://10.133.0.38:8643/healthz": dial tcp 10.133.0.38:8643: connect: connection refused
11:17:41 (x6)	kserve-ci-e2e-test	kubelet	success-200-isvc-bc774-predictor-5b6b775cdd-29slw	Unhealthy	Readiness probe failed: dial tcp 10.133.0.40:8080: connect: connection refused
11:17:48 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-bc774-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:17:48 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-bc774-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:17:51 (x8)	kserve-ci-e2e-test	kubelet	error-404-isvc-bc774-predictor-8547545bdf-ktmgb	Unhealthy	Readiness probe failed: dial tcp 10.133.0.41:8080: connect: connection refused
11:17:51	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-bc774	InferenceServiceReady	InferenceService [success-200-isvc-bc774] is Ready
11:17:51 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-bc774	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
11:18:00 (x6)	kserve-ci-e2e-test	kubelet	splitter-graph-cdb7a-ffd5b4495-dstlk	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
11:18:01	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-bc774	InferenceServiceReady	InferenceService [error-404-isvc-bc774] is Ready
11:18:01 (x5)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-bc774	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
11:18:03	kserve-ci-e2e-test	kubelet	switch-graph-bc774-7778877fb9-g5794	Created	Created container: switch-graph-bc774
11:18:03	kserve-ci-e2e-test	kubelet	switch-graph-bc774-7778877fb9-g5794	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:0e1a169e211c4cdcd19b5d9ae2b04ed590c1bcb2f57424f5c6e7d0d0f2faa1f0" already present on machine
11:18:03	kserve-ci-e2e-test	replicaset-controller	switch-graph-bc774-7778877fb9	SuccessfulCreate	Created pod: switch-graph-bc774-7778877fb9-g5794
11:18:03	kserve-ci-e2e-test	InferenceGraphController	switch-graph-bc774	InferenceGraphReady	InferenceGraph [switch-graph-bc774] is Ready
11:18:03	kserve-ci-e2e-test	kubelet	switch-graph-bc774-7778877fb9-g5794	Started	Started container switch-graph-bc774
11:18:03	kserve-ci-e2e-test	multus	switch-graph-bc774-7778877fb9-g5794	AddedInterface	Add eth0 [10.134.0.35/23] from ovn-kubernetes
11:18:03	kserve-ci-e2e-test	deployment-controller	switch-graph-bc774	ScalingReplicaSet	Scaled up replica set switch-graph-bc774-7778877fb9 from 0 to 1
11:18:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-d1ffd-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:18:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-d1ffd-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:18:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-d1ffd-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:18:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-d1ffd-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:18:18 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-bc774-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:18:18 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-bc774-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:18:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-bc774-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:18:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-bc774-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:18:21 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-d1ffd	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
11:18:21	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-d1ffd	InferenceServiceReady	InferenceService [success-200-isvc-d1ffd] is Ready
11:18:21 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-d1ffd	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
11:18:21	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-d1ffd	InferenceServiceReady	InferenceService [error-404-isvc-d1ffd] is Ready
11:18:33	kserve-ci-e2e-test	InferenceGraphController	splitter-graph-d1ffd	InferenceGraphReady	InferenceGraph [splitter-graph-d1ffd] is Ready
11:18:33	kserve-ci-e2e-test	kubelet	splitter-graph-d1ffd-7d9bf8b7bd-962wv	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:0e1a169e211c4cdcd19b5d9ae2b04ed590c1bcb2f57424f5c6e7d0d0f2faa1f0" already present on machine
11:18:33 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-bc774	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:18:33 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-bc774	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:18:33	kserve-ci-e2e-test	deployment-controller	splitter-graph-d1ffd	ScalingReplicaSet	Scaled up replica set splitter-graph-d1ffd-7d9bf8b7bd from 0 to 1
11:18:33	kserve-ci-e2e-test	replicaset-controller	splitter-graph-d1ffd-7d9bf8b7bd	SuccessfulCreate	Created pod: splitter-graph-d1ffd-7d9bf8b7bd-962wv
11:18:33	kserve-ci-e2e-test	multus	splitter-graph-d1ffd-7d9bf8b7bd-962wv	AddedInterface	Add eth0 [10.134.0.36/23] from ovn-kubernetes
11:18:34	kserve-ci-e2e-test	kubelet	splitter-graph-d1ffd-7d9bf8b7bd-962wv	Started	Started container splitter-graph-d1ffd
11:18:34	kserve-ci-e2e-test	kubelet	splitter-graph-d1ffd-7d9bf8b7bd-962wv	Created	Created container: splitter-graph-d1ffd
11:18:48 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-d1ffd-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:18:48 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-d1ffd-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:18:48 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-d1ffd-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:18:48 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-d1ffd-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:19:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	splitter-graph-d1ffd	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:19:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	splitter-graph-d1ffd	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:26:48	kserve-ci-e2e-test	kubelet	success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf	Killing	Stopping container kserve-container
11:26:48	kserve-ci-e2e-test	kubelet	error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2	Killing	Stopping container kube-rbac-proxy
11:26:48	kserve-ci-e2e-test	kubelet	splitter-graph-d1ffd-7d9bf8b7bd-962wv	Killing	Stopping container splitter-graph-d1ffd
11:26:48	kserve-ci-e2e-test	kubelet	success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf	Killing	Stopping container kube-rbac-proxy
11:26:48	kserve-ci-e2e-test	kubelet	error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2	Killing	Stopping container kserve-container
11:26:51 (x7)	kserve-ci-e2e-test	kubelet	success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf	Unhealthy	Readiness probe failed: dial tcp 10.133.0.42:8080: connect: connection refused
11:26:51 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2	Unhealthy	Readiness probe failed: dial tcp 10.133.0.43:8080: connect: connection refused
11:26:51	kserve-ci-e2e-test	kubelet	error-404-isvc-d1ffd-predictor-bb8d46565-r7hp2	Unhealthy	Readiness probe failed: Get "https://10.133.0.43:8643/healthz": dial tcp 10.133.0.43:8643: connect: connection refused
11:26:51	kserve-ci-e2e-test	kubelet	success-200-isvc-d1ffd-predictor-759c7cbb88-dbdcf	Unhealthy	Readiness probe failed: Get "https://10.133.0.42:8643/healthz": dial tcp 10.133.0.42:8643: connect: connection refused
11:27:15 (x6)	kserve-ci-e2e-test	kubelet	splitter-graph-d1ffd-7d9bf8b7bd-962wv	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
11:34:22	kserve-ci-e2e-test	kubelet	success-200-isvc-bc774-predictor-5b6b775cdd-29slw	Killing	Stopping container kserve-container
11:34:22	kserve-ci-e2e-test	kubelet	success-200-isvc-bc774-predictor-5b6b775cdd-29slw	Killing	Stopping container kube-rbac-proxy
11:34:22	kserve-ci-e2e-test	kubelet	error-404-isvc-bc774-predictor-8547545bdf-ktmgb	Killing	Stopping container kube-rbac-proxy
11:34:22	kserve-ci-e2e-test	kubelet	error-404-isvc-bc774-predictor-8547545bdf-ktmgb	Killing	Stopping container kserve-container
11:34:22	kserve-ci-e2e-test	kubelet	switch-graph-bc774-7778877fb9-g5794	Killing	Stopping container switch-graph-bc774
11:34:50 (x6)	kserve-ci-e2e-test	kubelet	switch-graph-bc774-7778877fb9-g5794	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503