Events

Time	Namespace	Component	RelatedObject	Reason	Message
13:29:19	kserve-ci-e2e-test		switch-graph-acf44-5486c59c4d-rtpd8	Scheduled	Successfully assigned kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8 to ip-10-0-133-122.ec2.internal
13:50:25	kserve-ci-e2e-test		success-200-isvc-03d0f-predictor-75bbcf79f5-hs9hm	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-03d0f-predictor-75bbcf79f5-hs9hm to ip-10-0-131-238.ec2.internal
13:39:24	kserve-ci-e2e-test		success-200-isvc-2bc9c-predictor-59ff898c87-zfgzl	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-2bc9c-predictor-59ff898c87-zfgzl to ip-10-0-131-238.ec2.internal
13:49:50	kserve-ci-e2e-test		success-200-isvc-7f184-predictor-5f7fcfbcbf-747l7	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-7f184-predictor-5f7fcfbcbf-747l7 to ip-10-0-131-238.ec2.internal
13:49:15	kserve-ci-e2e-test		success-200-isvc-9633b-predictor-774c89fdf5-kzkcr	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-9633b-predictor-774c89fdf5-kzkcr to ip-10-0-131-238.ec2.internal
13:27:39	kserve-ci-e2e-test		success-200-isvc-acf44-predictor-76c47c8649-p5q4f	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-acf44-predictor-76c47c8649-p5q4f to ip-10-0-131-238.ec2.internal
13:29:30	kserve-ci-e2e-test		error-404-isvc-383ed-predictor-6fbb59466b-2252r	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r to ip-10-0-133-122.ec2.internal
13:27:40	kserve-ci-e2e-test		isvc-sklearn-graph-1-predictor-6f5c857d79-k8xjk	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-k8xjk to ip-10-0-131-238.ec2.internal
13:50:15	kserve-ci-e2e-test		splitter-graph-9633b-986484644-tkj85	Scheduled	Successfully assigned kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85 to ip-10-0-133-122.ec2.internal
13:38:45	kserve-ci-e2e-test		success-200-isvc-6c5e8-predictor-866c9df647-kc54f	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-6c5e8-predictor-866c9df647-kc54f to ip-10-0-131-238.ec2.internal
13:40:55	kserve-ci-e2e-test		ensemble-graph-e2bd2-fc854767f-spdpt	Scheduled	Successfully assigned kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt to ip-10-0-133-122.ec2.internal
13:40:35	kserve-ci-e2e-test		error-404-isvc-7b15c-predictor-6c577469b4-7vqmd	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd to ip-10-0-133-122.ec2.internal
13:49:50	kserve-ci-e2e-test		error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk to ip-10-0-133-122.ec2.internal
13:30:10	kserve-ci-e2e-test		success-200-isvc-f369f-predictor-7957879bc7-pvbxz	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-f369f-predictor-7957879bc7-pvbxz to ip-10-0-131-238.ec2.internal
13:49:15	kserve-ci-e2e-test		error-404-isvc-9633b-predictor-6db75d69f8-hx5s5	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5 to ip-10-0-133-122.ec2.internal
13:39:45	kserve-ci-e2e-test		ensemble-graph-6c5e8-5fd6dd9659-55l9b	Scheduled	Successfully assigned kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b to ip-10-0-133-122.ec2.internal
13:29:30	kserve-ci-e2e-test		success-200-isvc-383ed-predictor-55fbb9d4ff-mc9t2	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-383ed-predictor-55fbb9d4ff-mc9t2 to ip-10-0-131-238.ec2.internal
13:30:10	kserve-ci-e2e-test		error-404-isvc-f369f-predictor-784754f556-n6j5x	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x to ip-10-0-133-122.ec2.internal
13:27:40	kserve-ci-e2e-test		isvc-sklearn-graph-2-predictor-6fb9776d89-b7dzn	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-b7dzn to ip-10-0-131-238.ec2.internal
13:51:25	kserve-ci-e2e-test		splitter-graph-03d0f-7b76878d5c-rzw2c	Scheduled	Successfully assigned kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c to ip-10-0-133-122.ec2.internal
13:31:10	kserve-ci-e2e-test		sequence-graph-f369f-d9b69b6db-2jn9z	Scheduled	Successfully assigned kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z to ip-10-0-133-122.ec2.internal
13:39:55	kserve-ci-e2e-test		success-200-isvc-e2bd2-predictor-6bcb66fb47-n57qc	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-e2bd2-predictor-6bcb66fb47-n57qc to ip-10-0-131-238.ec2.internal
13:39:55	kserve-ci-e2e-test		error-404-isvc-e2bd2-predictor-677686d8bb-mxljv	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv to ip-10-0-133-122.ec2.internal
13:41:35	kserve-ci-e2e-test		sequence-graph-7b15c-657959bb75-qgx8p	Scheduled	Successfully assigned kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p to ip-10-0-133-122.ec2.internal
13:40:24	kserve-ci-e2e-test		sequence-graph-2bc9c-6db49dd9b7-dfmqj	Scheduled	Successfully assigned kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj to ip-10-0-133-122.ec2.internal
13:50:25	kserve-ci-e2e-test		error-404-isvc-03d0f-predictor-59b6876b74-9wdb4	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4 to ip-10-0-133-122.ec2.internal
13:29:59	kserve-ci-e2e-test		model-chainer-9d455c95-bnm99	Scheduled	Successfully assigned kserve-ci-e2e-test/model-chainer-9d455c95-bnm99 to ip-10-0-133-122.ec2.internal
13:30:30	kserve-ci-e2e-test		switch-graph-383ed-79cc5cfd9b-cjs9c	Scheduled	Successfully assigned kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c to ip-10-0-133-122.ec2.internal
13:27:40	kserve-ci-e2e-test		error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h to ip-10-0-133-122.ec2.internal
13:40:35	kserve-ci-e2e-test		success-200-isvc-7b15c-predictor-6cdf8ff9bf-rrrzc	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-7b15c-predictor-6cdf8ff9bf-rrrzc to ip-10-0-131-238.ec2.internal
13:38:45	kserve-ci-e2e-test		error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd to ip-10-0-133-122.ec2.internal
13:39:24	kserve-ci-e2e-test		error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9 to ip-10-0-133-122.ec2.internal
13:27:40	kserve-ci-e2e-test		isvc-xgboost-graph-predictor-555fc69d56-p6mls	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-p6mls to ip-10-0-131-238.ec2.internal
13:50:50	kserve-ci-e2e-test		switch-graph-7f184-7676f88dd9-z4frx	Scheduled	Successfully assigned kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx to ip-10-0-133-122.ec2.internal
13:27:39	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-acf44-predictor-76c47c8649	SuccessfulCreate	Created pod: success-200-isvc-acf44-predictor-76c47c8649-p5q4f
13:27:39	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-acf44	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-acf44": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-acf44": the object has been modified; please apply your changes to the latest version and try again
13:27:39	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-graph-1-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-graph-1-predictor-6f5c857d79 from 0 to 1
13:27:39	kserve-ci-e2e-test	deployment-controller	success-200-isvc-acf44-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-acf44-predictor-76c47c8649 from 0 to 1
13:27:39	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-acf44	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-acf44": the object has been modified; please apply your changes to the latest version and try again
13:27:40	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-graph-predictor-555fc69d56	SuccessfulCreate	Created pod: isvc-xgboost-graph-predictor-555fc69d56-p6mls
13:27:40	kserve-ci-e2e-test	multus	error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h	AddedInterface	Add eth0 [10.134.0.29/23] from ovn-kubernetes
13:27:40	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-p6mls	Created	Created container: storage-initializer
13:27:40	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-p6mls	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:27:40	kserve-ci-e2e-test	multus	isvc-xgboost-graph-predictor-555fc69d56-p6mls	AddedInterface	Add eth0 [10.132.0.33/23] from ovn-kubernetes
13:27:40	kserve-ci-e2e-test	kubelet	error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h	Pulling	Pulling image "quay.io/opendatahub/error-404-isvc:odh-pr-1683"
13:27:40	kserve-ci-e2e-test	kubelet	success-200-isvc-acf44-predictor-76c47c8649-p5q4f	Pulling	Pulling image "quay.io/opendatahub/success-200-isvc:odh-pr-1683"
13:27:40	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-p6mls	Started	Started container storage-initializer
13:27:40	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-graph-2-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-graph-2-predictor-6fb9776d89 from 0 to 1
13:27:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-6f5c857d79-k8xjk	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
13:27:40	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-graph-2-predictor-6fb9776d89	SuccessfulCreate	Created pod: isvc-sklearn-graph-2-predictor-6fb9776d89-b7dzn
13:27:40	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6fb9776d89-b7dzn	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
13:27:40	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-acf44-predictor-7bdc57d7d4	SuccessfulCreate	Created pod: error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h
13:27:40	kserve-ci-e2e-test	deployment-controller	error-404-isvc-acf44-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-acf44-predictor-7bdc57d7d4 from 0 to 1
13:27:40	kserve-ci-e2e-test	multus	success-200-isvc-acf44-predictor-76c47c8649-p5q4f	AddedInterface	Add eth0 [10.132.0.31/23] from ovn-kubernetes
13:27:40	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-graph-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-graph-predictor-555fc69d56 from 0 to 1
13:27:40	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-graph-1-predictor-6f5c857d79	SuccessfulCreate	Created pod: isvc-sklearn-graph-1-predictor-6f5c857d79-k8xjk
13:27:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-6f5c857d79-k8xjk	Created	Created container: storage-initializer
13:27:41	kserve-ci-e2e-test	multus	isvc-sklearn-graph-1-predictor-6f5c857d79-k8xjk	AddedInterface	Add eth0 [10.132.0.32/23] from ovn-kubernetes
13:27:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6fb9776d89-b7dzn	Started	Started container storage-initializer
13:27:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6fb9776d89-b7dzn	Created	Created container: storage-initializer
13:27:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6fb9776d89-b7dzn	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:27:41	kserve-ci-e2e-test	multus	isvc-sklearn-graph-2-predictor-6fb9776d89-b7dzn	AddedInterface	Add eth0 [10.132.0.34/23] from ovn-kubernetes
13:27:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-6f5c857d79-k8xjk	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine
13:27:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-6f5c857d79-k8xjk	Started	Started container storage-initializer
13:27:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-6f5c857d79-k8xjk	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1683"
13:27:46	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6fb9776d89-b7dzn	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1683"
13:27:46	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-p6mls	Pulling	Pulling image "kserve/xgbserver:latest"
13:27:52	kserve-ci-e2e-test	kubelet	error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h	Created	Created container: kserve-container
13:27:52	kserve-ci-e2e-test	kubelet	error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
13:27:52	kserve-ci-e2e-test	kubelet	error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h	Pulled	Successfully pulled image "quay.io/opendatahub/error-404-isvc:odh-pr-1683" in 12.142s (12.142s including waiting). Image size: 1334623835 bytes.
13:27:52	kserve-ci-e2e-test	kubelet	error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h	Started	Started container kserve-container
13:27:55	kserve-ci-e2e-test	kubelet	error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h	Started	Started container kube-rbac-proxy
13:27:55	kserve-ci-e2e-test	kubelet	error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h	Created	Created container: kube-rbac-proxy
13:27:55	kserve-ci-e2e-test	kubelet	error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.289s (2.289s including waiting). Image size: 211946088 bytes.
13:28:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6fb9776d89-b7dzn	Created	Created container: kserve-container
13:28:02	kserve-ci-e2e-test	kubelet	success-200-isvc-acf44-predictor-76c47c8649-p5q4f	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
13:28:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-6f5c857d79-k8xjk	Created	Created container: kserve-container
13:28:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-6f5c857d79-k8xjk	Started	Started container kserve-container
13:28:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-6f5c857d79-k8xjk	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
13:28:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6fb9776d89-b7dzn	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
13:28:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-6f5c857d79-k8xjk	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1683" in 16.389s (16.389s including waiting). Image size: 1558595490 bytes.
13:28:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6fb9776d89-b7dzn	Started	Started container kserve-container
13:28:02	kserve-ci-e2e-test	kubelet	success-200-isvc-acf44-predictor-76c47c8649-p5q4f	Pulled	Successfully pulled image "quay.io/opendatahub/success-200-isvc:odh-pr-1683" in 22.193s (22.194s including waiting). Image size: 1335613021 bytes.
13:28:02	kserve-ci-e2e-test	kubelet	success-200-isvc-acf44-predictor-76c47c8649-p5q4f	Created	Created container: kserve-container
13:28:02	kserve-ci-e2e-test	kubelet	success-200-isvc-acf44-predictor-76c47c8649-p5q4f	Started	Started container kserve-container
13:28:02	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6fb9776d89-b7dzn	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1683" in 16.374s (16.374s including waiting). Image size: 1558595490 bytes.
13:28:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-6f5c857d79-k8xjk	Created	Created container: kube-rbac-proxy
13:28:06	kserve-ci-e2e-test	kubelet	success-200-isvc-acf44-predictor-76c47c8649-p5q4f	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 4.124s (4.124s including waiting). Image size: 211946088 bytes.
13:28:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-6f5c857d79-k8xjk	Started	Started container kube-rbac-proxy
13:28:06	kserve-ci-e2e-test	kubelet	success-200-isvc-acf44-predictor-76c47c8649-p5q4f	Created	Created container: kube-rbac-proxy
13:28:06	kserve-ci-e2e-test	kubelet	success-200-isvc-acf44-predictor-76c47c8649-p5q4f	Started	Started container kube-rbac-proxy
13:28:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-6f5c857d79-k8xjk	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 4.054s (4.054s including waiting). Image size: 211946088 bytes.
13:28:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6fb9776d89-b7dzn	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 4.077s (4.077s including waiting). Image size: 211946088 bytes.
13:28:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6fb9776d89-b7dzn	Created	Created container: kube-rbac-proxy
13:28:06	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6fb9776d89-b7dzn	Started	Started container kube-rbac-proxy
13:28:10 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:28:10 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:28:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-p6mls	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 30.535s (30.535s including waiting). Image size: 1305293048 bytes.
13:28:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-p6mls	Created	Created container: kserve-container
13:28:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-p6mls	Started	Started container kserve-container
13:28:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-p6mls	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
13:28:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-p6mls	Created	Created container: kube-rbac-proxy
13:28:16	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-p6mls	Started	Started container kube-rbac-proxy
13:28:25 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-acf44-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:28:25 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-acf44-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:28:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-acf44-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:28:39 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-acf44-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:28:40 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:28:40 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:28:42 (x5)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-acf44	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
13:28:42	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-acf44	InferenceServiceReady	InferenceService [error-404-isvc-acf44] is Ready
13:28:55 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:28:55 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:29:03	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-acf44	InferenceServiceReady	InferenceService [success-200-isvc-acf44] is Ready
13:29:03 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-acf44	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
13:29:10 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-acf44-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:29:10 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-acf44-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:29:17 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6fb9776d89-b7dzn	Unhealthy	Readiness probe failed: dial tcp 10.132.0.34:8080: connect: connection refused
13:29:19	kserve-ci-e2e-test	deployment-controller	switch-graph-acf44	ScalingReplicaSet	Scaled up replica set switch-graph-acf44-5486c59c4d from 0 to 1
13:29:19	kserve-ci-e2e-test	replicaset-controller	switch-graph-acf44-5486c59c4d	SuccessfulCreate	Created pod: switch-graph-acf44-5486c59c4d-rtpd8
13:29:19	kserve-ci-e2e-test	InferenceGraphController	switch-graph-acf44	InferenceGraphReady	InferenceGraph [switch-graph-acf44] is Ready
13:29:19	kserve-ci-e2e-test	InferenceGraphController	switch-graph-acf44	UpdateFailed	Failed to update status for InferenceGraph "switch-graph-acf44": Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "switch-graph-acf44": the object has been modified; please apply your changes to the latest version and try again
13:29:19	kserve-ci-e2e-test	InferenceGraphController	switch-graph-acf44	InternalError	fails to update InferenceGraph status: Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "switch-graph-acf44": the object has been modified; please apply your changes to the latest version and try again
13:29:20	kserve-ci-e2e-test	multus	switch-graph-acf44-5486c59c4d-rtpd8	AddedInterface	Add eth0 [10.134.0.30/23] from ovn-kubernetes
13:29:20	kserve-ci-e2e-test	kubelet	switch-graph-acf44-5486c59c4d-rtpd8	Pulling	Pulling image "quay.io/opendatahub/kserve-router@sha256:9f703da9f2f5afd108b6816a57feadccff9213d4d8437a63e732fa3fbaa3fa9d"
13:29:20	kserve-ci-e2e-test	kubelet	switch-graph-acf44-5486c59c4d-rtpd8	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "switch-graph-acf44-serving-cert" not found
13:29:23	kserve-ci-e2e-test	kubelet	switch-graph-acf44-5486c59c4d-rtpd8	Started	Started container switch-graph-acf44
13:29:23	kserve-ci-e2e-test	kubelet	switch-graph-acf44-5486c59c4d-rtpd8	Created	Created container: switch-graph-acf44
13:29:23	kserve-ci-e2e-test	kubelet	switch-graph-acf44-5486c59c4d-rtpd8	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:9f703da9f2f5afd108b6816a57feadccff9213d4d8437a63e732fa3fbaa3fa9d" in 2.401s (2.401s including waiting). Image size: 218070105 bytes.
13:29:24 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-6f5c857d79-k8xjk	Unhealthy	Readiness probe failed: dial tcp 10.132.0.32:8080: connect: connection refused
13:29:24 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-acf44-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:29:24 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-acf44-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:29:24 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-graph	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
13:29:24	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-graph	InferenceServiceReady	InferenceService [isvc-xgboost-graph] is Ready
13:29:27	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-2	InferenceServiceReady	InferenceService [isvc-sklearn-graph-2] is Ready
13:29:27 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
13:29:29	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-1	InferenceServiceReady	InferenceService [isvc-sklearn-graph-1] is Ready
13:29:29 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
13:29:30	kserve-ci-e2e-test	kubelet	success-200-isvc-383ed-predictor-55fbb9d4ff-mc9t2	Created	Created container: kube-rbac-proxy
13:29:30	kserve-ci-e2e-test	kubelet	error-404-isvc-383ed-predictor-6fbb59466b-2252r	Started	Started container kube-rbac-proxy
13:29:30	kserve-ci-e2e-test	kubelet	error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h	Killing	Stopping container kserve-container
13:29:30	kserve-ci-e2e-test	kubelet	error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h	Killing	Stopping container kube-rbac-proxy
13:29:30	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-383ed	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-383ed": the object has been modified; please apply your changes to the latest version and try again
13:29:30	kserve-ci-e2e-test	kubelet	success-200-isvc-383ed-predictor-55fbb9d4ff-mc9t2	Started	Started container kserve-container
13:29:30	kserve-ci-e2e-test	kubelet	success-200-isvc-383ed-predictor-55fbb9d4ff-mc9t2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
13:29:30	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-383ed	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-383ed": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-383ed": the object has been modified; please apply your changes to the latest version and try again
13:29:30	kserve-ci-e2e-test	kubelet	switch-graph-acf44-5486c59c4d-rtpd8	Killing	Stopping container switch-graph-acf44
13:29:30	kserve-ci-e2e-test	kubelet	success-200-isvc-acf44-predictor-76c47c8649-p5q4f	Killing	Stopping container kserve-container
13:29:30	kserve-ci-e2e-test	kubelet	success-200-isvc-383ed-predictor-55fbb9d4ff-mc9t2	Started	Started container kube-rbac-proxy
13:29:30	kserve-ci-e2e-test	deployment-controller	error-404-isvc-383ed-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-383ed-predictor-6fbb59466b from 0 to 1
13:29:30	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-383ed-predictor-6fbb59466b	SuccessfulCreate	Created pod: error-404-isvc-383ed-predictor-6fbb59466b-2252r
13:29:30	kserve-ci-e2e-test	kubelet	error-404-isvc-383ed-predictor-6fbb59466b-2252r	Created	Created container: kube-rbac-proxy
13:29:30	kserve-ci-e2e-test	kubelet	error-404-isvc-383ed-predictor-6fbb59466b-2252r	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
13:29:30	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-383ed-predictor-55fbb9d4ff	SuccessfulCreate	Created pod: success-200-isvc-383ed-predictor-55fbb9d4ff-mc9t2
13:29:30	kserve-ci-e2e-test	deployment-controller	success-200-isvc-383ed-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-383ed-predictor-55fbb9d4ff from 0 to 1
13:29:30	kserve-ci-e2e-test	kubelet	error-404-isvc-383ed-predictor-6fbb59466b-2252r	Started	Started container kserve-container
13:29:30	kserve-ci-e2e-test	kubelet	error-404-isvc-383ed-predictor-6fbb59466b-2252r	Created	Created container: kserve-container
13:29:30	kserve-ci-e2e-test	kubelet	success-200-isvc-383ed-predictor-55fbb9d4ff-mc9t2	Created	Created container: kserve-container
13:29:30	kserve-ci-e2e-test	kubelet	error-404-isvc-383ed-predictor-6fbb59466b-2252r	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1683" already present on machine
13:29:30	kserve-ci-e2e-test	kubelet	success-200-isvc-383ed-predictor-55fbb9d4ff-mc9t2	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1683" already present on machine
13:29:30	kserve-ci-e2e-test	multus	error-404-isvc-383ed-predictor-6fbb59466b-2252r	AddedInterface	Add eth0 [10.134.0.31/23] from ovn-kubernetes
13:29:30	kserve-ci-e2e-test	multus	success-200-isvc-383ed-predictor-55fbb9d4ff-mc9t2	AddedInterface	Add eth0 [10.132.0.35/23] from ovn-kubernetes
13:29:30	kserve-ci-e2e-test	kubelet	success-200-isvc-acf44-predictor-76c47c8649-p5q4f	Killing	Stopping container kube-rbac-proxy
13:29:32 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h	Unhealthy	Readiness probe failed: dial tcp 10.134.0.29:8080: connect: connection refused
13:29:32	kserve-ci-e2e-test	kubelet	error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h	Unhealthy	Readiness probe failed: Get "https://10.134.0.29:8643/healthz": dial tcp 10.134.0.29:8643: connect: connection refused
13:29:33	kserve-ci-e2e-test	kubelet	success-200-isvc-acf44-predictor-76c47c8649-p5q4f	Unhealthy	Readiness probe failed: Get "https://10.132.0.31:8643/healthz": dial tcp 10.132.0.31:8643: connect: connection refused
13:29:33 (x8)	kserve-ci-e2e-test	kubelet	success-200-isvc-acf44-predictor-76c47c8649-p5q4f	Unhealthy	Readiness probe failed: dial tcp 10.132.0.31:8080: connect: connection refused
13:29:40 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:29:40 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:29:40 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:29:40 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:29:55 (x7)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:29:55 (x7)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:29:59 (x6)	kserve-ci-e2e-test	kubelet	switch-graph-acf44-5486c59c4d-rtpd8	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
13:29:59	kserve-ci-e2e-test	replicaset-controller	model-chainer-9d455c95	SuccessfulCreate	Created pod: model-chainer-9d455c95-bnm99
13:29:59	kserve-ci-e2e-test	deployment-controller	model-chainer	ScalingReplicaSet	Scaled up replica set model-chainer-9d455c95 from 0 to 1
13:29:59	kserve-ci-e2e-test	InferenceGraphController	model-chainer	InferenceGraphReady	InferenceGraph [model-chainer] is Ready
13:30:00	kserve-ci-e2e-test	kubelet	model-chainer-9d455c95-bnm99	Started	Started container model-chainer
13:30:00	kserve-ci-e2e-test	kubelet	model-chainer-9d455c95-bnm99	Created	Created container: model-chainer
13:30:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-383ed-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:30:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-383ed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:30:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-383ed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:30:00	kserve-ci-e2e-test	multus	model-chainer-9d455c95-bnm99	AddedInterface	Add eth0 [10.134.0.32/23] from ovn-kubernetes
13:30:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-383ed-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:30:00	kserve-ci-e2e-test	kubelet	model-chainer-9d455c95-bnm99	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:9f703da9f2f5afd108b6816a57feadccff9213d4d8437a63e732fa3fbaa3fa9d" already present on machine
13:30:08 (x6)	kserve-ci-e2e-test	kubelet	error-404-isvc-383ed-predictor-6fbb59466b-2252r	Unhealthy	Readiness probe failed: dial tcp 10.134.0.31:8080: connect: connection refused
13:30:09	kserve-ci-e2e-test	kubelet	model-chainer-9d455c95-bnm99	Killing	Stopping container model-chainer
13:30:10	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-p6mls	Killing	Stopping container kserve-container
13:30:10	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6fb9776d89-b7dzn	Killing	Stopping container kserve-container
13:30:10	kserve-ci-e2e-test	kubelet	error-404-isvc-f369f-predictor-784754f556-n6j5x	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1683" already present on machine
13:30:10	kserve-ci-e2e-test	multus	error-404-isvc-f369f-predictor-784754f556-n6j5x	AddedInterface	Add eth0 [10.134.0.33/23] from ovn-kubernetes
13:30:10	kserve-ci-e2e-test	kubelet	error-404-isvc-f369f-predictor-784754f556-n6j5x	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
13:30:10	kserve-ci-e2e-test	kubelet	error-404-isvc-f369f-predictor-784754f556-n6j5x	Created	Created container: kube-rbac-proxy
13:30:10	kserve-ci-e2e-test	kubelet	error-404-isvc-f369f-predictor-784754f556-n6j5x	Started	Started container kube-rbac-proxy
13:30:10	kserve-ci-e2e-test	deployment-controller	success-200-isvc-f369f-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-f369f-predictor-7957879bc7 from 0 to 1
13:30:10	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-f369f-predictor-7957879bc7	SuccessfulCreate	Created pod: success-200-isvc-f369f-predictor-7957879bc7-pvbxz
13:30:10	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-f369f-predictor-784754f556	SuccessfulCreate	Created pod: error-404-isvc-f369f-predictor-784754f556-n6j5x
13:30:10	kserve-ci-e2e-test	kubelet	error-404-isvc-f369f-predictor-784754f556-n6j5x	Started	Started container kserve-container
13:30:10	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-p6mls	Killing	Stopping container kube-rbac-proxy
13:30:10	kserve-ci-e2e-test	kubelet	error-404-isvc-f369f-predictor-784754f556-n6j5x	Created	Created container: kserve-container
13:30:10	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-6f5c857d79-k8xjk	Killing	Stopping container kserve-container
13:30:10	kserve-ci-e2e-test	kubelet	success-200-isvc-f369f-predictor-7957879bc7-pvbxz	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "success-200-isvc-f369f-predictor-serving-cert" not found
13:30:10	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6fb9776d89-b7dzn	Killing	Stopping container kube-rbac-proxy
13:30:10	kserve-ci-e2e-test	deployment-controller	error-404-isvc-f369f-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-f369f-predictor-784754f556 from 0 to 1
13:30:10	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-6f5c857d79-k8xjk	Killing	Stopping container kube-rbac-proxy
13:30:11	kserve-ci-e2e-test	kubelet	success-200-isvc-f369f-predictor-7957879bc7-pvbxz	Started	Started container kserve-container
13:30:11	kserve-ci-e2e-test	kubelet	success-200-isvc-f369f-predictor-7957879bc7-pvbxz	Created	Created container: kserve-container
13:30:11	kserve-ci-e2e-test	kubelet	success-200-isvc-f369f-predictor-7957879bc7-pvbxz	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
13:30:11	kserve-ci-e2e-test	kubelet	success-200-isvc-f369f-predictor-7957879bc7-pvbxz	Created	Created container: kube-rbac-proxy
13:30:11	kserve-ci-e2e-test	kubelet	success-200-isvc-f369f-predictor-7957879bc7-pvbxz	Started	Started container kube-rbac-proxy
13:30:11	kserve-ci-e2e-test	kubelet	success-200-isvc-f369f-predictor-7957879bc7-pvbxz	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1683" already present on machine
13:30:11	kserve-ci-e2e-test	multus	success-200-isvc-f369f-predictor-7957879bc7-pvbxz	AddedInterface	Add eth0 [10.132.0.36/23] from ovn-kubernetes
13:30:14	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-p6mls	Unhealthy	Readiness probe failed: Get "https://10.132.0.33:8643/healthz": dial tcp 10.132.0.33:8643: connect: connection refused
13:30:14 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-p6mls	Unhealthy	Readiness probe failed: dial tcp 10.132.0.33:8080: connect: connection refused
13:30:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6fb9776d89-b7dzn	Unhealthy	Readiness probe failed: Get "https://10.132.0.34:8643/healthz": dial tcp 10.132.0.34:8643: connect: connection refused
13:30:14	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-6f5c857d79-k8xjk	Unhealthy	Readiness probe failed: Get "https://10.132.0.32:8643/healthz": dial tcp 10.132.0.32:8643: connect: connection refused
13:30:17 (x6)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-383ed	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
13:30:17	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-383ed	InferenceServiceReady	InferenceService [success-200-isvc-383ed] is Ready
13:30:18	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-383ed	InferenceServiceReady	InferenceService [error-404-isvc-383ed] is Ready
13:30:18 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-383ed	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
13:30:30	kserve-ci-e2e-test	deployment-controller	switch-graph-383ed	ScalingReplicaSet	Scaled up replica set switch-graph-383ed-79cc5cfd9b from 0 to 1
13:30:30	kserve-ci-e2e-test	InferenceGraphController	switch-graph-383ed	InferenceGraphReady	InferenceGraph [switch-graph-383ed] is Ready
13:30:30	kserve-ci-e2e-test	replicaset-controller	switch-graph-383ed-79cc5cfd9b	SuccessfulCreate	Created pod: switch-graph-383ed-79cc5cfd9b-cjs9c
13:30:30	kserve-ci-e2e-test	kubelet	switch-graph-383ed-79cc5cfd9b-cjs9c	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "switch-graph-383ed-serving-cert" not found
13:30:31	kserve-ci-e2e-test	kubelet	switch-graph-383ed-79cc5cfd9b-cjs9c	Started	Started container switch-graph-383ed
13:30:31	kserve-ci-e2e-test	multus	switch-graph-383ed-79cc5cfd9b-cjs9c	AddedInterface	Add eth0 [10.134.0.34/23] from ovn-kubernetes
13:30:31	kserve-ci-e2e-test	kubelet	switch-graph-383ed-79cc5cfd9b-cjs9c	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:9f703da9f2f5afd108b6816a57feadccff9213d4d8437a63e732fa3fbaa3fa9d" already present on machine
13:30:31	kserve-ci-e2e-test	kubelet	switch-graph-383ed-79cc5cfd9b-cjs9c	Created	Created container: switch-graph-383ed
13:30:36 (x6)	kserve-ci-e2e-test	kubelet	model-chainer-9d455c95-bnm99	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
13:30:40 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-f369f-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:30:40 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-f369f-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:30:40 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-f369f-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:30:40 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-f369f-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:30:45 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-383ed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:30:45 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-383ed-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:30:48 (x6)	kserve-ci-e2e-test	kubelet	success-200-isvc-f369f-predictor-7957879bc7-pvbxz	Unhealthy	Readiness probe failed: dial tcp 10.132.0.36:8080: connect: connection refused
13:30:56	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-f369f	InferenceServiceReady	InferenceService [error-404-isvc-f369f] is Ready
13:30:56 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-f369f	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
13:30:59 (x6)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-f369f	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
13:30:59	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-f369f	InferenceServiceReady	InferenceService [success-200-isvc-f369f] is Ready
13:31:00 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-383ed-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:31:00 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-383ed-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:31:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-383ed	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:31:00 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-383ed	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:31:10 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-f369f-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:31:10	kserve-ci-e2e-test	deployment-controller	sequence-graph-f369f	ScalingReplicaSet	Scaled up replica set sequence-graph-f369f-d9b69b6db from 0 to 1
13:31:10	kserve-ci-e2e-test	InferenceGraphController	sequence-graph-f369f	InferenceGraphReady	InferenceGraph [sequence-graph-f369f] is Ready
13:31:10 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-f369f-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:31:10	kserve-ci-e2e-test	replicaset-controller	sequence-graph-f369f-d9b69b6db	SuccessfulCreate	Created pod: sequence-graph-f369f-d9b69b6db-2jn9z
13:31:10	kserve-ci-e2e-test	kubelet	sequence-graph-f369f-d9b69b6db-2jn9z	Started	Started container sequence-graph-f369f
13:31:10	kserve-ci-e2e-test	kubelet	sequence-graph-f369f-d9b69b6db-2jn9z	Created	Created container: sequence-graph-f369f
13:31:10	kserve-ci-e2e-test	kubelet	sequence-graph-f369f-d9b69b6db-2jn9z	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:9f703da9f2f5afd108b6816a57feadccff9213d4d8437a63e732fa3fbaa3fa9d" already present on machine
13:31:10	kserve-ci-e2e-test	multus	sequence-graph-f369f-d9b69b6db-2jn9z	AddedInterface	Add eth0 [10.134.0.35/23] from ovn-kubernetes
13:31:25 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-f369f-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:31:25 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-f369f-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:31:40 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-f369f	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:31:40 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-f369f	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:38:45	kserve-ci-e2e-test	kubelet	error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd	Created	Created container: kserve-container
13:38:45	kserve-ci-e2e-test	kubelet	error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1683" already present on machine
13:38:45	kserve-ci-e2e-test	kubelet	error-404-isvc-383ed-predictor-6fbb59466b-2252r	Killing	Stopping container kserve-container
13:38:45	kserve-ci-e2e-test	kubelet	success-200-isvc-383ed-predictor-55fbb9d4ff-mc9t2	Killing	Stopping container kserve-container
13:38:45	kserve-ci-e2e-test	deployment-controller	success-200-isvc-6c5e8-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-6c5e8-predictor-866c9df647 from 0 to 1
13:38:45	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-6c5e8-predictor-866c9df647	SuccessfulCreate	Created pod: success-200-isvc-6c5e8-predictor-866c9df647-kc54f
13:38:45	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-6c5e8	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-6c5e8": the object has been modified; please apply your changes to the latest version and try again
13:38:45	kserve-ci-e2e-test	kubelet	error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd	Started	Started container kserve-container
13:38:45	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-6c5e8	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-6c5e8": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-6c5e8": the object has been modified; please apply your changes to the latest version and try again
13:38:45	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-6c5e8	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-6c5e8": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-6c5e8": the object has been modified; please apply your changes to the latest version and try again
13:38:45	kserve-ci-e2e-test	kubelet	switch-graph-383ed-79cc5cfd9b-cjs9c	Killing	Stopping container switch-graph-383ed
13:38:45	kserve-ci-e2e-test	kubelet	error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd	Created	Created container: kube-rbac-proxy
13:38:45	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-6c5e8-predictor-bbf5dc8fd	SuccessfulCreate	Created pod: error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd
13:38:45	kserve-ci-e2e-test	kubelet	success-200-isvc-383ed-predictor-55fbb9d4ff-mc9t2	Killing	Stopping container kube-rbac-proxy
13:38:45	kserve-ci-e2e-test	kubelet	error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
13:38:45	kserve-ci-e2e-test	kubelet	error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd	Started	Started container kube-rbac-proxy
13:38:45	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-6c5e8	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-6c5e8": the object has been modified; please apply your changes to the latest version and try again
13:38:45	kserve-ci-e2e-test	multus	error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd	AddedInterface	Add eth0 [10.134.0.36/23] from ovn-kubernetes
13:38:45	kserve-ci-e2e-test	kubelet	success-200-isvc-6c5e8-predictor-866c9df647-kc54f	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "success-200-isvc-6c5e8-predictor-serving-cert" not found
13:38:45	kserve-ci-e2e-test	kubelet	error-404-isvc-383ed-predictor-6fbb59466b-2252r	Killing	Stopping container kube-rbac-proxy
13:38:45	kserve-ci-e2e-test	deployment-controller	error-404-isvc-6c5e8-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-6c5e8-predictor-bbf5dc8fd from 0 to 1
13:38:46	kserve-ci-e2e-test	multus	success-200-isvc-6c5e8-predictor-866c9df647-kc54f	AddedInterface	Add eth0 [10.132.0.37/23] from ovn-kubernetes
13:38:46	kserve-ci-e2e-test	kubelet	success-200-isvc-6c5e8-predictor-866c9df647-kc54f	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1683" already present on machine
13:38:46	kserve-ci-e2e-test	kubelet	success-200-isvc-6c5e8-predictor-866c9df647-kc54f	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
13:38:46	kserve-ci-e2e-test	kubelet	success-200-isvc-6c5e8-predictor-866c9df647-kc54f	Created	Created container: kserve-container
13:38:46	kserve-ci-e2e-test	kubelet	success-200-isvc-6c5e8-predictor-866c9df647-kc54f	Created	Created container: kube-rbac-proxy
13:38:46	kserve-ci-e2e-test	kubelet	success-200-isvc-6c5e8-predictor-866c9df647-kc54f	Started	Started container kserve-container
13:38:46	kserve-ci-e2e-test	kubelet	success-200-isvc-6c5e8-predictor-866c9df647-kc54f	Started	Started container kube-rbac-proxy
13:38:47	kserve-ci-e2e-test	kubelet	success-200-isvc-383ed-predictor-55fbb9d4ff-mc9t2	Unhealthy	Readiness probe failed: Get "https://10.132.0.35:8643/healthz": dial tcp 10.132.0.35:8643: connect: connection refused
13:38:47 (x7)	kserve-ci-e2e-test	kubelet	success-200-isvc-383ed-predictor-55fbb9d4ff-mc9t2	Unhealthy	Readiness probe failed: dial tcp 10.132.0.35:8080: connect: connection refused
13:39:12 (x6)	kserve-ci-e2e-test	kubelet	switch-graph-383ed-79cc5cfd9b-cjs9c	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
13:39:15 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-6c5e8-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:39:15 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-6c5e8-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:39:15 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-6c5e8-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:39:15 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-6c5e8-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:39:22 (x6)	kserve-ci-e2e-test	kubelet	success-200-isvc-6c5e8-predictor-866c9df647-kc54f	Unhealthy	Readiness probe failed: dial tcp 10.132.0.37:8080: connect: connection refused
13:39:22 (x6)	kserve-ci-e2e-test	kubelet	error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd	Unhealthy	Readiness probe failed: dial tcp 10.134.0.36:8080: connect: connection refused
13:39:24	kserve-ci-e2e-test	kubelet	error-404-isvc-f369f-predictor-784754f556-n6j5x	Killing	Stopping container kube-rbac-proxy
13:39:24	kserve-ci-e2e-test	kubelet	error-404-isvc-f369f-predictor-784754f556-n6j5x	Killing	Stopping container kserve-container
13:39:24	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-2bc9c-predictor-54f54f9cbb	SuccessfulCreate	Created pod: error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9
13:39:24	kserve-ci-e2e-test	deployment-controller	error-404-isvc-2bc9c-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-2bc9c-predictor-54f54f9cbb from 0 to 1
13:39:24	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-2bc9c	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-2bc9c": the object has been modified; please apply your changes to the latest version and try again
13:39:24	kserve-ci-e2e-test	deployment-controller	success-200-isvc-2bc9c-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-2bc9c-predictor-59ff898c87 from 0 to 1
13:39:24	kserve-ci-e2e-test	kubelet	sequence-graph-f369f-d9b69b6db-2jn9z	Killing	Stopping container sequence-graph-f369f
13:39:24	kserve-ci-e2e-test	kubelet	success-200-isvc-f369f-predictor-7957879bc7-pvbxz	Killing	Stopping container kube-rbac-proxy
13:39:24	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-2bc9c	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-2bc9c": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-2bc9c": the object has been modified; please apply your changes to the latest version and try again
13:39:24	kserve-ci-e2e-test	kubelet	success-200-isvc-f369f-predictor-7957879bc7-pvbxz	Killing	Stopping container kserve-container
13:39:24	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-2bc9c-predictor-59ff898c87	SuccessfulCreate	Created pod: success-200-isvc-2bc9c-predictor-59ff898c87-zfgzl
13:39:25	kserve-ci-e2e-test	multus	success-200-isvc-2bc9c-predictor-59ff898c87-zfgzl	AddedInterface	Add eth0 [10.132.0.38/23] from ovn-kubernetes
13:39:25	kserve-ci-e2e-test	kubelet	error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "error-404-isvc-2bc9c-predictor-serving-cert" not found
13:39:25	kserve-ci-e2e-test	kubelet	success-200-isvc-2bc9c-predictor-59ff898c87-zfgzl	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1683" already present on machine
13:39:25	kserve-ci-e2e-test	kubelet	success-200-isvc-2bc9c-predictor-59ff898c87-zfgzl	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "success-200-isvc-2bc9c-predictor-serving-cert" not found
13:39:26	kserve-ci-e2e-test	kubelet	error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1683" already present on machine
13:39:26	kserve-ci-e2e-test	kubelet	error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9	Created	Created container: kube-rbac-proxy
13:39:26	kserve-ci-e2e-test	multus	error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9	AddedInterface	Add eth0 [10.134.0.37/23] from ovn-kubernetes
13:39:26	kserve-ci-e2e-test	kubelet	success-200-isvc-2bc9c-predictor-59ff898c87-zfgzl	Created	Created container: kserve-container
13:39:26	kserve-ci-e2e-test	kubelet	error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9	Created	Created container: kserve-container
13:39:26	kserve-ci-e2e-test	kubelet	error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9	Started	Started container kserve-container
13:39:26	kserve-ci-e2e-test	kubelet	error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
13:39:26	kserve-ci-e2e-test	kubelet	success-200-isvc-2bc9c-predictor-59ff898c87-zfgzl	Started	Started container kube-rbac-proxy
13:39:26	kserve-ci-e2e-test	kubelet	error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9	Started	Started container kube-rbac-proxy
13:39:26	kserve-ci-e2e-test	kubelet	success-200-isvc-2bc9c-predictor-59ff898c87-zfgzl	Started	Started container kserve-container
13:39:26	kserve-ci-e2e-test	kubelet	success-200-isvc-2bc9c-predictor-59ff898c87-zfgzl	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
13:39:26 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-f369f-predictor-784754f556-n6j5x	Unhealthy	Readiness probe failed: dial tcp 10.134.0.33:8080: connect: connection refused
13:39:26	kserve-ci-e2e-test	kubelet	error-404-isvc-f369f-predictor-784754f556-n6j5x	Unhealthy	Readiness probe failed: Get "https://10.134.0.33:8643/healthz": dial tcp 10.134.0.33:8643: connect: connection refused
13:39:26	kserve-ci-e2e-test	kubelet	success-200-isvc-2bc9c-predictor-59ff898c87-zfgzl	Created	Created container: kube-rbac-proxy
13:39:32	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-6c5e8	InferenceServiceReady	InferenceService [success-200-isvc-6c5e8] is Ready
13:39:32 (x6)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-6c5e8	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
13:39:33	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-6c5e8	InferenceServiceReady	InferenceService [error-404-isvc-6c5e8] is Ready
13:39:33 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-6c5e8	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
13:39:45	kserve-ci-e2e-test	replicaset-controller	ensemble-graph-6c5e8-5fd6dd9659	SuccessfulCreate	Created pod: ensemble-graph-6c5e8-5fd6dd9659-55l9b
13:39:45	kserve-ci-e2e-test	kubelet	ensemble-graph-6c5e8-5fd6dd9659-55l9b	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:9f703da9f2f5afd108b6816a57feadccff9213d4d8437a63e732fa3fbaa3fa9d" already present on machine
13:39:45	kserve-ci-e2e-test	multus	ensemble-graph-6c5e8-5fd6dd9659-55l9b	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
13:39:45 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-6c5e8-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:39:45 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-6c5e8-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:39:45	kserve-ci-e2e-test	kubelet	ensemble-graph-6c5e8-5fd6dd9659-55l9b	Created	Created container: ensemble-graph-6c5e8
13:39:45	kserve-ci-e2e-test	kubelet	ensemble-graph-6c5e8-5fd6dd9659-55l9b	Started	Started container ensemble-graph-6c5e8
13:39:45	kserve-ci-e2e-test	deployment-controller	ensemble-graph-6c5e8	ScalingReplicaSet	Scaled up replica set ensemble-graph-6c5e8-5fd6dd9659 from 0 to 1
13:39:45	kserve-ci-e2e-test	InferenceGraphController	ensemble-graph-6c5e8	InferenceGraphReady	InferenceGraph [ensemble-graph-6c5e8] is Ready
13:39:45 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-6c5e8-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:39:45 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-6c5e8-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:39:52 (x6)	kserve-ci-e2e-test	kubelet	sequence-graph-f369f-d9b69b6db-2jn9z	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
13:39:54 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-2bc9c-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:39:54 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-2bc9c-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:39:54 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-2bc9c-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:39:54 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-2bc9c-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:39:55	kserve-ci-e2e-test	deployment-controller	success-200-isvc-e2bd2-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-e2bd2-predictor-6bcb66fb47 from 0 to 1
13:39:55	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-e2bd2	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-e2bd2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-e2bd2": the object has been modified; please apply your changes to the latest version and try again
13:39:55	kserve-ci-e2e-test	kubelet	success-200-isvc-e2bd2-predictor-6bcb66fb47-n57qc	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "success-200-isvc-e2bd2-predictor-serving-cert" not found
13:39:55	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-e2bd2-predictor-6bcb66fb47	SuccessfulCreate	Created pod: success-200-isvc-e2bd2-predictor-6bcb66fb47-n57qc
13:39:55	kserve-ci-e2e-test	kubelet	ensemble-graph-6c5e8-5fd6dd9659-55l9b	Killing	Stopping container ensemble-graph-6c5e8
13:39:55	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-e2bd2-predictor-677686d8bb	SuccessfulCreate	Created pod: error-404-isvc-e2bd2-predictor-677686d8bb-mxljv
13:39:55	kserve-ci-e2e-test	kubelet	success-200-isvc-6c5e8-predictor-866c9df647-kc54f	Killing	Stopping container kube-rbac-proxy
13:39:55	kserve-ci-e2e-test	kubelet	success-200-isvc-6c5e8-predictor-866c9df647-kc54f	Killing	Stopping container kserve-container
13:39:55	kserve-ci-e2e-test	kubelet	error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd	Killing	Stopping container kserve-container
13:39:55	kserve-ci-e2e-test	kubelet	error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd	Killing	Stopping container kube-rbac-proxy
13:39:55	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-e2bd2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-e2bd2": the object has been modified; please apply your changes to the latest version and try again
13:39:55	kserve-ci-e2e-test	kubelet	error-404-isvc-e2bd2-predictor-677686d8bb-mxljv	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "error-404-isvc-e2bd2-predictor-serving-cert" not found
13:39:55	kserve-ci-e2e-test	deployment-controller	error-404-isvc-e2bd2-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-e2bd2-predictor-677686d8bb from 0 to 1
13:39:56	kserve-ci-e2e-test	multus	error-404-isvc-e2bd2-predictor-677686d8bb-mxljv	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
13:39:56	kserve-ci-e2e-test	kubelet	success-200-isvc-e2bd2-predictor-6bcb66fb47-n57qc	Started	Started container kube-rbac-proxy
13:39:56	kserve-ci-e2e-test	kubelet	error-404-isvc-e2bd2-predictor-677686d8bb-mxljv	Started	Started container kserve-container
13:39:56	kserve-ci-e2e-test	kubelet	error-404-isvc-e2bd2-predictor-677686d8bb-mxljv	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
13:39:56	kserve-ci-e2e-test	kubelet	error-404-isvc-e2bd2-predictor-677686d8bb-mxljv	Created	Created container: kube-rbac-proxy
13:39:56	kserve-ci-e2e-test	kubelet	error-404-isvc-e2bd2-predictor-677686d8bb-mxljv	Started	Started container kube-rbac-proxy
13:39:56	kserve-ci-e2e-test	multus	success-200-isvc-e2bd2-predictor-6bcb66fb47-n57qc	AddedInterface	Add eth0 [10.132.0.39/23] from ovn-kubernetes
13:39:56	kserve-ci-e2e-test	kubelet	error-404-isvc-e2bd2-predictor-677686d8bb-mxljv	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1683" already present on machine
13:39:56	kserve-ci-e2e-test	kubelet	success-200-isvc-e2bd2-predictor-6bcb66fb47-n57qc	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1683" already present on machine
13:39:56	kserve-ci-e2e-test	kubelet	success-200-isvc-e2bd2-predictor-6bcb66fb47-n57qc	Created	Created container: kserve-container
13:39:56	kserve-ci-e2e-test	kubelet	success-200-isvc-e2bd2-predictor-6bcb66fb47-n57qc	Started	Started container kserve-container
13:39:56	kserve-ci-e2e-test	kubelet	error-404-isvc-e2bd2-predictor-677686d8bb-mxljv	Created	Created container: kserve-container
13:39:56	kserve-ci-e2e-test	kubelet	success-200-isvc-e2bd2-predictor-6bcb66fb47-n57qc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
13:39:56	kserve-ci-e2e-test	kubelet	success-200-isvc-e2bd2-predictor-6bcb66fb47-n57qc	Created	Created container: kube-rbac-proxy
13:39:57	kserve-ci-e2e-test	kubelet	error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd	Unhealthy	Readiness probe failed: Get "https://10.134.0.36:8643/healthz": dial tcp 10.134.0.36:8643: connect: connection refused
13:39:57	kserve-ci-e2e-test	kubelet	success-200-isvc-6c5e8-predictor-866c9df647-kc54f	Unhealthy	Readiness probe failed: Get "https://10.132.0.37:8643/healthz": dial tcp 10.132.0.37:8643: connect: connection refused
13:40:03 (x6)	kserve-ci-e2e-test	kubelet	success-200-isvc-2bc9c-predictor-59ff898c87-zfgzl	Unhealthy	Readiness probe failed: dial tcp 10.132.0.38:8080: connect: connection refused
13:40:04 (x6)	kserve-ci-e2e-test	kubelet	error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9	Unhealthy	Readiness probe failed: dial tcp 10.134.0.37:8080: connect: connection refused
13:40:13	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-2bc9c	InferenceServiceReady	InferenceService [success-200-isvc-2bc9c] is Ready
13:40:13 (x6)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-2bc9c	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
13:40:14	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-2bc9c	InferenceServiceReady	InferenceService [error-404-isvc-2bc9c] is Ready
13:40:14 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-2bc9c	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
13:40:22 (x6)	kserve-ci-e2e-test	kubelet	ensemble-graph-6c5e8-5fd6dd9659-55l9b	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
13:40:24	kserve-ci-e2e-test	replicaset-controller	sequence-graph-2bc9c-6db49dd9b7	SuccessfulCreate	Created pod: sequence-graph-2bc9c-6db49dd9b7-dfmqj
13:40:24	kserve-ci-e2e-test	deployment-controller	sequence-graph-2bc9c	ScalingReplicaSet	Scaled up replica set sequence-graph-2bc9c-6db49dd9b7 from 0 to 1
13:40:24 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-2bc9c-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:40:24 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-2bc9c-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:40:24	kserve-ci-e2e-test	InferenceGraphController	sequence-graph-2bc9c	InferenceGraphReady	InferenceGraph [sequence-graph-2bc9c] is Ready
13:40:25 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-e2bd2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:40:25 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-2bc9c-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:40:25 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-e2bd2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:40:25 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-e2bd2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:40:25 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-2bc9c-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:40:25	kserve-ci-e2e-test	kubelet	sequence-graph-2bc9c-6db49dd9b7-dfmqj	Started	Started container sequence-graph-2bc9c
13:40:25	kserve-ci-e2e-test	kubelet	sequence-graph-2bc9c-6db49dd9b7-dfmqj	Created	Created container: sequence-graph-2bc9c
13:40:25 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-e2bd2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:40:25	kserve-ci-e2e-test	kubelet	sequence-graph-2bc9c-6db49dd9b7-dfmqj	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:9f703da9f2f5afd108b6816a57feadccff9213d4d8437a63e732fa3fbaa3fa9d" already present on machine
13:40:25	kserve-ci-e2e-test	multus	sequence-graph-2bc9c-6db49dd9b7-dfmqj	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
13:40:33 (x6)	kserve-ci-e2e-test	kubelet	success-200-isvc-e2bd2-predictor-6bcb66fb47-n57qc	Unhealthy	Readiness probe failed: dial tcp 10.132.0.39:8080: connect: connection refused
13:40:33 (x6)	kserve-ci-e2e-test	kubelet	error-404-isvc-e2bd2-predictor-677686d8bb-mxljv	Unhealthy	Readiness probe failed: dial tcp 10.134.0.39:8080: connect: connection refused
13:40:34	kserve-ci-e2e-test	kubelet	sequence-graph-2bc9c-6db49dd9b7-dfmqj	Killing	Stopping container sequence-graph-2bc9c
13:40:35	kserve-ci-e2e-test	kubelet	error-404-isvc-7b15c-predictor-6c577469b4-7vqmd	Created	Created container: kserve-container
13:40:35	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-7b15c	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-7b15c": the object has been modified; please apply your changes to the latest version and try again
13:40:35	kserve-ci-e2e-test	kubelet	success-200-isvc-7b15c-predictor-6cdf8ff9bf-rrrzc	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1683" already present on machine
13:40:35	kserve-ci-e2e-test	multus	success-200-isvc-7b15c-predictor-6cdf8ff9bf-rrrzc	AddedInterface	Add eth0 [10.132.0.40/23] from ovn-kubernetes
13:40:35	kserve-ci-e2e-test	kubelet	success-200-isvc-7b15c-predictor-6cdf8ff9bf-rrrzc	Started	Started container kserve-container
13:40:35	kserve-ci-e2e-test	kubelet	success-200-isvc-7b15c-predictor-6cdf8ff9bf-rrrzc	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
13:40:35	kserve-ci-e2e-test	kubelet	success-200-isvc-7b15c-predictor-6cdf8ff9bf-rrrzc	Created	Created container: kube-rbac-proxy
13:40:35	kserve-ci-e2e-test	kubelet	success-200-isvc-7b15c-predictor-6cdf8ff9bf-rrrzc	Started	Started container kube-rbac-proxy
13:40:35	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-7b15c-predictor-6cdf8ff9bf	SuccessfulCreate	Created pod: success-200-isvc-7b15c-predictor-6cdf8ff9bf-rrrzc
13:40:35	kserve-ci-e2e-test	deployment-controller	success-200-isvc-7b15c-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-7b15c-predictor-6cdf8ff9bf from 0 to 1
13:40:35	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-7b15c	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-7b15c": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-7b15c": the object has been modified; please apply your changes to the latest version and try again
13:40:35	kserve-ci-e2e-test	kubelet	error-404-isvc-7b15c-predictor-6c577469b4-7vqmd	Started	Started container kserve-container
13:40:35	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-7b15c-predictor-6c577469b4	SuccessfulCreate	Created pod: error-404-isvc-7b15c-predictor-6c577469b4-7vqmd
13:40:35	kserve-ci-e2e-test	kubelet	error-404-isvc-7b15c-predictor-6c577469b4-7vqmd	Started	Started container kube-rbac-proxy
13:40:35	kserve-ci-e2e-test	kubelet	success-200-isvc-7b15c-predictor-6cdf8ff9bf-rrrzc	Created	Created container: kserve-container
13:40:35	kserve-ci-e2e-test	deployment-controller	error-404-isvc-7b15c-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-7b15c-predictor-6c577469b4 from 0 to 1
13:40:35	kserve-ci-e2e-test	kubelet	error-404-isvc-7b15c-predictor-6c577469b4-7vqmd	Created	Created container: kube-rbac-proxy
13:40:35	kserve-ci-e2e-test	kubelet	error-404-isvc-7b15c-predictor-6c577469b4-7vqmd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
13:40:35	kserve-ci-e2e-test	kubelet	error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9	Killing	Stopping container kserve-container
13:40:35	kserve-ci-e2e-test	kubelet	error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9	Killing	Stopping container kube-rbac-proxy
13:40:35	kserve-ci-e2e-test	multus	error-404-isvc-7b15c-predictor-6c577469b4-7vqmd	AddedInterface	Add eth0 [10.134.0.41/23] from ovn-kubernetes
13:40:35	kserve-ci-e2e-test	kubelet	error-404-isvc-7b15c-predictor-6c577469b4-7vqmd	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1683" already present on machine
13:40:35	kserve-ci-e2e-test	kubelet	success-200-isvc-2bc9c-predictor-59ff898c87-zfgzl	Killing	Stopping container kserve-container
13:40:35	kserve-ci-e2e-test	kubelet	success-200-isvc-2bc9c-predictor-59ff898c87-zfgzl	Killing	Stopping container kube-rbac-proxy
13:40:43	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-e2bd2	InferenceServiceReady	InferenceService [error-404-isvc-e2bd2] is Ready
13:40:43 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-e2bd2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
13:40:43	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-e2bd2	InferenceServiceReady	InferenceService [success-200-isvc-e2bd2] is Ready
13:40:43 (x6)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-e2bd2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
13:40:55	kserve-ci-e2e-test	replicaset-controller	ensemble-graph-e2bd2-fc854767f	SuccessfulCreate	Created pod: ensemble-graph-e2bd2-fc854767f-spdpt
13:40:55	kserve-ci-e2e-test	InferenceGraphController	ensemble-graph-e2bd2	InferenceGraphReady	InferenceGraph [ensemble-graph-e2bd2] is Ready
13:40:55 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-e2bd2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:40:55 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-e2bd2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:40:55	kserve-ci-e2e-test	deployment-controller	ensemble-graph-e2bd2	ScalingReplicaSet	Scaled up replica set ensemble-graph-e2bd2-fc854767f from 0 to 1
13:40:55	kserve-ci-e2e-test	kubelet	ensemble-graph-e2bd2-fc854767f-spdpt	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "ensemble-graph-e2bd2-serving-cert" not found
13:40:56	kserve-ci-e2e-test	kubelet	ensemble-graph-e2bd2-fc854767f-spdpt	Started	Started container ensemble-graph-e2bd2
13:40:56	kserve-ci-e2e-test	kubelet	ensemble-graph-e2bd2-fc854767f-spdpt	Created	Created container: ensemble-graph-e2bd2
13:40:56	kserve-ci-e2e-test	kubelet	ensemble-graph-e2bd2-fc854767f-spdpt	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:9f703da9f2f5afd108b6816a57feadccff9213d4d8437a63e732fa3fbaa3fa9d" already present on machine
13:40:56	kserve-ci-e2e-test	multus	ensemble-graph-e2bd2-fc854767f-spdpt	AddedInterface	Add eth0 [10.134.0.42/23] from ovn-kubernetes
13:41:02 (x6)	kserve-ci-e2e-test	kubelet	sequence-graph-2bc9c-6db49dd9b7-dfmqj	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
13:41:10 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-e2bd2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:41:10 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-e2bd2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:41:10	kserve-ci-e2e-test	horizontal-pod-autoscaler	ensemble-graph-e2bd2	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:41:10	kserve-ci-e2e-test	horizontal-pod-autoscaler	ensemble-graph-e2bd2	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:41:13 (x6)	kserve-ci-e2e-test	kubelet	error-404-isvc-7b15c-predictor-6c577469b4-7vqmd	Unhealthy	Readiness probe failed: dial tcp 10.134.0.41:8080: connect: connection refused
13:41:20 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-7b15c-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:41:20 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-7b15c-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:41:20 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-7b15c-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:41:20 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-7b15c-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:41:23 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-7b15c	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
13:41:23	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-7b15c	InferenceServiceReady	InferenceService [error-404-isvc-7b15c] is Ready
13:41:23 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-7b15c	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
13:41:23	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-7b15c	InferenceServiceReady	InferenceService [success-200-isvc-7b15c] is Ready
13:41:25	kserve-ci-e2e-test	horizontal-pod-autoscaler	ensemble-graph-e2bd2	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:41:25	kserve-ci-e2e-test	horizontal-pod-autoscaler	ensemble-graph-e2bd2	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:41:35	kserve-ci-e2e-test	InferenceGraphController	sequence-graph-7b15c	InferenceGraphReady	InferenceGraph [sequence-graph-7b15c] is Ready
13:41:35	kserve-ci-e2e-test	deployment-controller	sequence-graph-7b15c	ScalingReplicaSet	Scaled up replica set sequence-graph-7b15c-657959bb75 from 0 to 1
13:41:35	kserve-ci-e2e-test	replicaset-controller	sequence-graph-7b15c-657959bb75	SuccessfulCreate	Created pod: sequence-graph-7b15c-657959bb75-qgx8p
13:41:35	kserve-ci-e2e-test	kubelet	sequence-graph-7b15c-657959bb75-qgx8p	Started	Started container sequence-graph-7b15c
13:41:35	kserve-ci-e2e-test	kubelet	sequence-graph-7b15c-657959bb75-qgx8p	Created	Created container: sequence-graph-7b15c
13:41:35	kserve-ci-e2e-test	kubelet	sequence-graph-7b15c-657959bb75-qgx8p	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:9f703da9f2f5afd108b6816a57feadccff9213d4d8437a63e732fa3fbaa3fa9d" already present on machine
13:41:35	kserve-ci-e2e-test	multus	sequence-graph-7b15c-657959bb75-qgx8p	AddedInterface	Add eth0 [10.134.0.43/23] from ovn-kubernetes
13:41:50 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-7b15c-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:41:50 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-7b15c-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:41:50 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-7b15c-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:41:50 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-7b15c-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:42:20 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-7b15c	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:42:20 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-7b15c	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:49:15	kserve-ci-e2e-test	kubelet	error-404-isvc-9633b-predictor-6db75d69f8-hx5s5	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1683" already present on machine
13:49:15	kserve-ci-e2e-test	kubelet	error-404-isvc-9633b-predictor-6db75d69f8-hx5s5	Started	Started container kserve-container
13:49:15	kserve-ci-e2e-test	kubelet	ensemble-graph-e2bd2-fc854767f-spdpt	Killing	Stopping container ensemble-graph-e2bd2
13:49:15	kserve-ci-e2e-test	multus	success-200-isvc-9633b-predictor-774c89fdf5-kzkcr	AddedInterface	Add eth0 [10.132.0.41/23] from ovn-kubernetes
13:49:15	kserve-ci-e2e-test	kubelet	success-200-isvc-9633b-predictor-774c89fdf5-kzkcr	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1683" already present on machine
13:49:15	kserve-ci-e2e-test	kubelet	success-200-isvc-9633b-predictor-774c89fdf5-kzkcr	Created	Created container: kserve-container
13:49:15	kserve-ci-e2e-test	kubelet	success-200-isvc-9633b-predictor-774c89fdf5-kzkcr	Started	Started container kserve-container
13:49:15	kserve-ci-e2e-test	kubelet	success-200-isvc-9633b-predictor-774c89fdf5-kzkcr	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
13:49:15	kserve-ci-e2e-test	kubelet	success-200-isvc-9633b-predictor-774c89fdf5-kzkcr	Created	Created container: kube-rbac-proxy
13:49:15	kserve-ci-e2e-test	kubelet	success-200-isvc-9633b-predictor-774c89fdf5-kzkcr	Started	Started container kube-rbac-proxy
13:49:15	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-9633b-predictor-774c89fdf5	SuccessfulCreate	Created pod: success-200-isvc-9633b-predictor-774c89fdf5-kzkcr
13:49:15	kserve-ci-e2e-test	multus	error-404-isvc-9633b-predictor-6db75d69f8-hx5s5	AddedInterface	Add eth0 [10.134.0.44/23] from ovn-kubernetes
13:49:15	kserve-ci-e2e-test	kubelet	error-404-isvc-9633b-predictor-6db75d69f8-hx5s5	Created	Created container: kserve-container
13:49:15	kserve-ci-e2e-test	kubelet	error-404-isvc-e2bd2-predictor-677686d8bb-mxljv	Killing	Stopping container kube-rbac-proxy
13:49:15	kserve-ci-e2e-test	kubelet	error-404-isvc-9633b-predictor-6db75d69f8-hx5s5	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
13:49:15	kserve-ci-e2e-test	kubelet	error-404-isvc-e2bd2-predictor-677686d8bb-mxljv	Killing	Stopping container kserve-container
13:49:15	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-9633b	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-9633b": the object has been modified; please apply your changes to the latest version and try again
13:49:15	kserve-ci-e2e-test	deployment-controller	success-200-isvc-9633b-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-9633b-predictor-774c89fdf5 from 0 to 1
13:49:15	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-9633b	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-9633b": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-9633b": the object has been modified; please apply your changes to the latest version and try again
13:49:15	kserve-ci-e2e-test	kubelet	success-200-isvc-e2bd2-predictor-6bcb66fb47-n57qc	Killing	Stopping container kube-rbac-proxy
13:49:15	kserve-ci-e2e-test	kubelet	success-200-isvc-e2bd2-predictor-6bcb66fb47-n57qc	Killing	Stopping container kserve-container
13:49:15	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-9633b-predictor-6db75d69f8	SuccessfulCreate	Created pod: error-404-isvc-9633b-predictor-6db75d69f8-hx5s5
13:49:15	kserve-ci-e2e-test	deployment-controller	error-404-isvc-9633b-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-9633b-predictor-6db75d69f8 from 0 to 1
13:49:15	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-9633b	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-9633b": the object has been modified; please apply your changes to the latest version and try again
13:49:15	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-9633b	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-9633b": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-9633b": the object has been modified; please apply your changes to the latest version and try again
13:49:16	kserve-ci-e2e-test	kubelet	error-404-isvc-9633b-predictor-6db75d69f8-hx5s5	Started	Started container kube-rbac-proxy
13:49:16	kserve-ci-e2e-test	kubelet	error-404-isvc-9633b-predictor-6db75d69f8-hx5s5	Created	Created container: kube-rbac-proxy
13:49:18	kserve-ci-e2e-test	kubelet	error-404-isvc-e2bd2-predictor-677686d8bb-mxljv	Unhealthy	Readiness probe failed: Get "https://10.134.0.39:8643/healthz": dial tcp 10.134.0.39:8643: connect: connection refused
13:49:30	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-9633b-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:49:30	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-9633b-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:49:43 (x6)	kserve-ci-e2e-test	kubelet	ensemble-graph-e2bd2-fc854767f-spdpt	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
13:49:45 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-9633b-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:49:45 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-9633b-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:49:49	kserve-ci-e2e-test	kubelet	sequence-graph-7b15c-657959bb75-qgx8p	Killing	Stopping container sequence-graph-7b15c
13:49:50	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-7f184	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-7f184": the object has been modified; please apply your changes to the latest version and try again
13:49:50	kserve-ci-e2e-test	deployment-controller	success-200-isvc-7f184-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-7f184-predictor-5f7fcfbcbf from 0 to 1
13:49:50	kserve-ci-e2e-test	kubelet	success-200-isvc-7b15c-predictor-6cdf8ff9bf-rrrzc	Killing	Stopping container kserve-container
13:49:50	kserve-ci-e2e-test	kubelet	success-200-isvc-7f184-predictor-5f7fcfbcbf-747l7	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "success-200-isvc-7f184-predictor-serving-cert" not found
13:49:50	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-7f184	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-7f184": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-7f184": the object has been modified; please apply your changes to the latest version and try again
13:49:50	kserve-ci-e2e-test	deployment-controller	error-404-isvc-7f184-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-7f184-predictor-5c9f6d84c6 from 0 to 1
13:49:50	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-7f184-predictor-5f7fcfbcbf	SuccessfulCreate	Created pod: success-200-isvc-7f184-predictor-5f7fcfbcbf-747l7
13:49:50	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-7f184	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-7f184": the object has been modified; please apply your changes to the latest version and try again
13:49:50	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-7f184-predictor-5c9f6d84c6	SuccessfulCreate	Created pod: error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk
13:49:50	kserve-ci-e2e-test	kubelet	error-404-isvc-7b15c-predictor-6c577469b4-7vqmd	Killing	Stopping container kserve-container
13:49:50	kserve-ci-e2e-test	kubelet	error-404-isvc-7b15c-predictor-6c577469b4-7vqmd	Killing	Stopping container kube-rbac-proxy
13:49:50	kserve-ci-e2e-test	kubelet	error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "error-404-isvc-7f184-predictor-serving-cert" not found
13:49:50	kserve-ci-e2e-test	kubelet	success-200-isvc-7b15c-predictor-6cdf8ff9bf-rrrzc	Killing	Stopping container kube-rbac-proxy
13:49:50	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-7f184	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-7f184": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-7f184": the object has been modified; please apply your changes to the latest version and try again
13:49:51	kserve-ci-e2e-test	kubelet	success-200-isvc-7f184-predictor-5f7fcfbcbf-747l7	Created	Created container: kube-rbac-proxy
13:49:51	kserve-ci-e2e-test	kubelet	error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk	Started	Started container kube-rbac-proxy
13:49:51	kserve-ci-e2e-test	kubelet	success-200-isvc-7f184-predictor-5f7fcfbcbf-747l7	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1683" already present on machine
13:49:51	kserve-ci-e2e-test	kubelet	success-200-isvc-7f184-predictor-5f7fcfbcbf-747l7	Created	Created container: kserve-container
13:49:51	kserve-ci-e2e-test	multus	error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk	AddedInterface	Add eth0 [10.134.0.45/23] from ovn-kubernetes
13:49:51	kserve-ci-e2e-test	kubelet	error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1683" already present on machine
13:49:51	kserve-ci-e2e-test	kubelet	error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk	Created	Created container: kserve-container
13:49:51	kserve-ci-e2e-test	kubelet	error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk	Started	Started container kserve-container
13:49:51	kserve-ci-e2e-test	kubelet	error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
13:49:51	kserve-ci-e2e-test	kubelet	error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk	Created	Created container: kube-rbac-proxy
13:49:51	kserve-ci-e2e-test	kubelet	success-200-isvc-7f184-predictor-5f7fcfbcbf-747l7	Started	Started container kserve-container
13:49:51	kserve-ci-e2e-test	multus	success-200-isvc-7f184-predictor-5f7fcfbcbf-747l7	AddedInterface	Add eth0 [10.132.0.42/23] from ovn-kubernetes
13:49:51	kserve-ci-e2e-test	kubelet	success-200-isvc-7f184-predictor-5f7fcfbcbf-747l7	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
13:49:51	kserve-ci-e2e-test	kubelet	success-200-isvc-7f184-predictor-5f7fcfbcbf-747l7	Started	Started container kube-rbac-proxy
13:49:52	kserve-ci-e2e-test	kubelet	success-200-isvc-7b15c-predictor-6cdf8ff9bf-rrrzc	Unhealthy	Readiness probe failed: Get "https://10.132.0.40:8643/healthz": dial tcp 10.132.0.40:8643: connect: connection refused
13:49:52 (x6)	kserve-ci-e2e-test	kubelet	error-404-isvc-9633b-predictor-6db75d69f8-hx5s5	Unhealthy	Readiness probe failed: dial tcp 10.134.0.44:8080: connect: connection refused
13:49:52 (x7)	kserve-ci-e2e-test	kubelet	success-200-isvc-7b15c-predictor-6cdf8ff9bf-rrrzc	Unhealthy	Readiness probe failed: dial tcp 10.132.0.40:8080: connect: connection refused
13:49:53	kserve-ci-e2e-test	kubelet	error-404-isvc-7b15c-predictor-6c577469b4-7vqmd	Unhealthy	Readiness probe failed: Get "https://10.134.0.41:8643/healthz": dial tcp 10.134.0.41:8643: connect: connection refused
13:49:53 (x6)	kserve-ci-e2e-test	kubelet	success-200-isvc-9633b-predictor-774c89fdf5-kzkcr	Unhealthy	Readiness probe failed: dial tcp 10.132.0.41:8080: connect: connection refused
13:49:54	kserve-ci-e2e-test	kubelet	error-404-isvc-7b15c-predictor-6c577469b4-7vqmd	Unhealthy	Readiness probe failed: dial tcp 10.134.0.41:8080: i/o timeout
13:50:02 (x5)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-9633b	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
13:50:02	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-9633b	InferenceServiceReady	InferenceService [error-404-isvc-9633b] is Ready
13:50:03 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-9633b	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
13:50:03	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-9633b	InferenceServiceReady	InferenceService [success-200-isvc-9633b] is Ready
13:50:15 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-9633b-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:50:15	kserve-ci-e2e-test	kubelet	splitter-graph-9633b-986484644-tkj85	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "splitter-graph-9633b-serving-cert" not found
13:50:15	kserve-ci-e2e-test	InferenceGraphController	splitter-graph-9633b	InferenceGraphReady	InferenceGraph [splitter-graph-9633b] is Ready
13:50:15	kserve-ci-e2e-test	deployment-controller	splitter-graph-9633b	ScalingReplicaSet	Scaled up replica set splitter-graph-9633b-986484644 from 0 to 1
13:50:15 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-9633b-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:50:15 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-9633b-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:50:15	kserve-ci-e2e-test	replicaset-controller	splitter-graph-9633b-986484644	SuccessfulCreate	Created pod: splitter-graph-9633b-986484644-tkj85
13:50:15 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-9633b-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:50:16	kserve-ci-e2e-test	multus	splitter-graph-9633b-986484644-tkj85	AddedInterface	Add eth0 [10.134.0.46/23] from ovn-kubernetes
13:50:16	kserve-ci-e2e-test	kubelet	splitter-graph-9633b-986484644-tkj85	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:9f703da9f2f5afd108b6816a57feadccff9213d4d8437a63e732fa3fbaa3fa9d" already present on machine
13:50:16 (x6)	kserve-ci-e2e-test	kubelet	sequence-graph-7b15c-657959bb75-qgx8p	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
13:50:16	kserve-ci-e2e-test	kubelet	splitter-graph-9633b-986484644-tkj85	Started	Started container splitter-graph-9633b
13:50:16	kserve-ci-e2e-test	kubelet	splitter-graph-9633b-986484644-tkj85	Created	Created container: splitter-graph-9633b
13:50:25	kserve-ci-e2e-test	kubelet	splitter-graph-9633b-986484644-tkj85	Killing	Stopping container splitter-graph-9633b
13:50:25	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-03d0f-predictor-75bbcf79f5	SuccessfulCreate	Created pod: success-200-isvc-03d0f-predictor-75bbcf79f5-hs9hm
13:50:25	kserve-ci-e2e-test	kubelet	error-404-isvc-03d0f-predictor-59b6876b74-9wdb4	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "error-404-isvc-03d0f-predictor-serving-cert" not found
13:50:25	kserve-ci-e2e-test	deployment-controller	error-404-isvc-03d0f-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-03d0f-predictor-59b6876b74 from 0 to 1
13:50:25	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-03d0f-predictor-59b6876b74	SuccessfulCreate	Created pod: error-404-isvc-03d0f-predictor-59b6876b74-9wdb4
13:50:25	kserve-ci-e2e-test	kubelet	success-200-isvc-9633b-predictor-774c89fdf5-kzkcr	Killing	Stopping container kube-rbac-proxy
13:50:25	kserve-ci-e2e-test	kubelet	error-404-isvc-9633b-predictor-6db75d69f8-hx5s5	Killing	Stopping container kserve-container
13:50:25	kserve-ci-e2e-test	kubelet	error-404-isvc-9633b-predictor-6db75d69f8-hx5s5	Killing	Stopping container kube-rbac-proxy
13:50:25	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-03d0f	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-03d0f": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-03d0f": the object has been modified; please apply your changes to the latest version and try again
13:50:25	kserve-ci-e2e-test	kubelet	success-200-isvc-9633b-predictor-774c89fdf5-kzkcr	Killing	Stopping container kserve-container
13:50:25	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-03d0f	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-03d0f": the object has been modified; please apply your changes to the latest version and try again
13:50:25	kserve-ci-e2e-test	deployment-controller	success-200-isvc-03d0f-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-03d0f-predictor-75bbcf79f5 from 0 to 1
13:50:26	kserve-ci-e2e-test	kubelet	error-404-isvc-03d0f-predictor-59b6876b74-9wdb4	Started	Started container kube-rbac-proxy
13:50:26	kserve-ci-e2e-test	kubelet	success-200-isvc-03d0f-predictor-75bbcf79f5-hs9hm	Created	Created container: kserve-container
13:50:26	kserve-ci-e2e-test	kubelet	error-404-isvc-03d0f-predictor-59b6876b74-9wdb4	Created	Created container: kserve-container
13:50:26	kserve-ci-e2e-test	kubelet	error-404-isvc-03d0f-predictor-59b6876b74-9wdb4	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1683" already present on machine
13:50:26	kserve-ci-e2e-test	multus	success-200-isvc-03d0f-predictor-75bbcf79f5-hs9hm	AddedInterface	Add eth0 [10.132.0.43/23] from ovn-kubernetes
13:50:26	kserve-ci-e2e-test	kubelet	success-200-isvc-03d0f-predictor-75bbcf79f5-hs9hm	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1683" already present on machine
13:50:26	kserve-ci-e2e-test	multus	error-404-isvc-03d0f-predictor-59b6876b74-9wdb4	AddedInterface	Add eth0 [10.134.0.47/23] from ovn-kubernetes
13:50:26	kserve-ci-e2e-test	kubelet	error-404-isvc-03d0f-predictor-59b6876b74-9wdb4	Started	Started container kserve-container
13:50:26	kserve-ci-e2e-test	kubelet	error-404-isvc-03d0f-predictor-59b6876b74-9wdb4	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
13:50:26	kserve-ci-e2e-test	kubelet	success-200-isvc-03d0f-predictor-75bbcf79f5-hs9hm	Started	Started container kube-rbac-proxy
13:50:26	kserve-ci-e2e-test	kubelet	success-200-isvc-03d0f-predictor-75bbcf79f5-hs9hm	Created	Created container: kube-rbac-proxy
13:50:26	kserve-ci-e2e-test	kubelet	success-200-isvc-03d0f-predictor-75bbcf79f5-hs9hm	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
13:50:26	kserve-ci-e2e-test	kubelet	error-404-isvc-03d0f-predictor-59b6876b74-9wdb4	Created	Created container: kube-rbac-proxy
13:50:26	kserve-ci-e2e-test	kubelet	success-200-isvc-03d0f-predictor-75bbcf79f5-hs9hm	Started	Started container kserve-container
13:50:27	kserve-ci-e2e-test	kubelet	error-404-isvc-9633b-predictor-6db75d69f8-hx5s5	Unhealthy	Readiness probe failed: Get "https://10.134.0.44:8643/healthz": dial tcp 10.134.0.44:8643: connect: connection refused
13:50:28	kserve-ci-e2e-test	kubelet	success-200-isvc-9633b-predictor-774c89fdf5-kzkcr	Unhealthy	Readiness probe failed: Get "https://10.132.0.41:8643/healthz": dial tcp 10.132.0.41:8643: connect: connection refused
13:50:28 (x6)	kserve-ci-e2e-test	kubelet	success-200-isvc-7f184-predictor-5f7fcfbcbf-747l7	Unhealthy	Readiness probe failed: dial tcp 10.132.0.42:8080: connect: connection refused
13:50:28 (x6)	kserve-ci-e2e-test	kubelet	error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk	Unhealthy	Readiness probe failed: dial tcp 10.134.0.45:8080: connect: connection refused
13:50:35 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-7f184-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:50:35 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-7f184-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:50:35 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-7f184-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:50:35 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-7f184-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:50:38	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-7f184	InferenceServiceReady	InferenceService [success-200-isvc-7f184] is Ready
13:50:38 (x6)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-7f184	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
13:50:38	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-7f184	InferenceServiceReady	InferenceService [error-404-isvc-7f184] is Ready
13:50:39 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-7f184	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
13:50:40	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-03d0f-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:50:40	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-03d0f-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:50:50	kserve-ci-e2e-test	replicaset-controller	switch-graph-7f184-7676f88dd9	SuccessfulCreate	Created pod: switch-graph-7f184-7676f88dd9-z4frx
13:50:50	kserve-ci-e2e-test	kubelet	switch-graph-7f184-7676f88dd9-z4frx	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "switch-graph-7f184-serving-cert" not found
13:50:50	kserve-ci-e2e-test	InferenceGraphController	switch-graph-7f184	InferenceGraphReady	InferenceGraph [switch-graph-7f184] is Ready
13:50:50	kserve-ci-e2e-test	deployment-controller	switch-graph-7f184	ScalingReplicaSet	Scaled up replica set switch-graph-7f184-7676f88dd9 from 0 to 1
13:50:51	kserve-ci-e2e-test	kubelet	switch-graph-7f184-7676f88dd9-z4frx	Created	Created container: switch-graph-7f184
13:50:51	kserve-ci-e2e-test	kubelet	switch-graph-7f184-7676f88dd9-z4frx	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:9f703da9f2f5afd108b6816a57feadccff9213d4d8437a63e732fa3fbaa3fa9d" already present on machine
13:50:51	kserve-ci-e2e-test	kubelet	switch-graph-7f184-7676f88dd9-z4frx	Started	Started container switch-graph-7f184
13:50:51	kserve-ci-e2e-test	multus	switch-graph-7f184-7676f88dd9-z4frx	AddedInterface	Add eth0 [10.134.0.48/23] from ovn-kubernetes
13:50:53 (x6)	kserve-ci-e2e-test	kubelet	splitter-graph-9633b-986484644-tkj85	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
13:50:55 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-03d0f-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:50:55 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-03d0f-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:51:03 (x6)	kserve-ci-e2e-test	kubelet	success-200-isvc-03d0f-predictor-75bbcf79f5-hs9hm	Unhealthy	Readiness probe failed: dial tcp 10.132.0.43:8080: connect: connection refused
13:51:05 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-7f184-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:51:05 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-7f184-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:51:05 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-7f184-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:51:05 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-7f184-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:51:13 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-03d0f	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
13:51:13	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-03d0f	InferenceServiceReady	InferenceService [error-404-isvc-03d0f] is Ready
13:51:13 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-03d0f	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
13:51:13	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-03d0f	InferenceServiceReady	InferenceService [success-200-isvc-03d0f] is Ready
13:51:25	kserve-ci-e2e-test	replicaset-controller	splitter-graph-03d0f-7b76878d5c	SuccessfulCreate	Created pod: splitter-graph-03d0f-7b76878d5c-rzw2c
13:51:25	kserve-ci-e2e-test	InferenceGraphController	splitter-graph-03d0f	InternalError	fails to update InferenceGraph status: Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "splitter-graph-03d0f": the object has been modified; please apply your changes to the latest version and try again
13:51:25 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-03d0f-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:51:25 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-03d0f-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:51:25	kserve-ci-e2e-test	deployment-controller	splitter-graph-03d0f	ScalingReplicaSet	Scaled up replica set splitter-graph-03d0f-7b76878d5c from 0 to 1
13:51:25	kserve-ci-e2e-test	InferenceGraphController	splitter-graph-03d0f	InferenceGraphReady	InferenceGraph [splitter-graph-03d0f] is Ready
13:51:25	kserve-ci-e2e-test	InferenceGraphController	splitter-graph-03d0f	UpdateFailed	Failed to update status for InferenceGraph "splitter-graph-03d0f": Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "splitter-graph-03d0f": the object has been modified; please apply your changes to the latest version and try again
13:51:26	kserve-ci-e2e-test	kubelet	splitter-graph-03d0f-7b76878d5c-rzw2c	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:9f703da9f2f5afd108b6816a57feadccff9213d4d8437a63e732fa3fbaa3fa9d" already present on machine
13:51:26	kserve-ci-e2e-test	kubelet	splitter-graph-03d0f-7b76878d5c-rzw2c	Created	Created container: splitter-graph-03d0f
13:51:26	kserve-ci-e2e-test	kubelet	splitter-graph-03d0f-7b76878d5c-rzw2c	Started	Started container splitter-graph-03d0f
13:51:26	kserve-ci-e2e-test	multus	splitter-graph-03d0f-7b76878d5c-rzw2c	AddedInterface	Add eth0 [10.134.0.49/23] from ovn-kubernetes
13:51:35 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-7f184	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:51:35 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-7f184	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:51:40 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-03d0f-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:51:40 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-03d0f-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
13:51:55 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	splitter-graph-03d0f	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:51:55 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	splitter-graph-03d0f	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
13:59:40	kserve-ci-e2e-test	kubelet	success-200-isvc-03d0f-predictor-75bbcf79f5-hs9hm	Killing	Stopping container kube-rbac-proxy
13:59:40	kserve-ci-e2e-test	kubelet	success-200-isvc-03d0f-predictor-75bbcf79f5-hs9hm	Killing	Stopping container kserve-container
13:59:40	kserve-ci-e2e-test	kubelet	error-404-isvc-03d0f-predictor-59b6876b74-9wdb4	Killing	Stopping container kube-rbac-proxy
13:59:40	kserve-ci-e2e-test	kubelet	error-404-isvc-03d0f-predictor-59b6876b74-9wdb4	Killing	Stopping container kserve-container
13:59:40	kserve-ci-e2e-test	kubelet	splitter-graph-03d0f-7b76878d5c-rzw2c	Killing	Stopping container splitter-graph-03d0f
13:59:43 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-03d0f-predictor-59b6876b74-9wdb4	Unhealthy	Readiness probe failed: dial tcp 10.134.0.47:8080: connect: connection refused
13:59:43	kserve-ci-e2e-test	kubelet	error-404-isvc-03d0f-predictor-59b6876b74-9wdb4	Unhealthy	Readiness probe failed: Get "https://10.134.0.47:8643/healthz": dial tcp 10.134.0.47:8643: connect: connection refused
14:00:07 (x6)	kserve-ci-e2e-test	kubelet	splitter-graph-03d0f-7b76878d5c-rzw2c	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
14:07:09	kserve-ci-e2e-test	kubelet	error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk	Killing	Stopping container kserve-container
14:07:09	kserve-ci-e2e-test	kubelet	success-200-isvc-7f184-predictor-5f7fcfbcbf-747l7	Killing	Stopping container kube-rbac-proxy
14:07:09	kserve-ci-e2e-test	kubelet	success-200-isvc-7f184-predictor-5f7fcfbcbf-747l7	Killing	Stopping container kserve-container
14:07:09	kserve-ci-e2e-test	kubelet	switch-graph-7f184-7676f88dd9-z4frx	Killing	Stopping container switch-graph-7f184
14:07:09	kserve-ci-e2e-test	kubelet	error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk	Killing	Stopping container kube-rbac-proxy
14:07:33 (x5)	kserve-ci-e2e-test	kubelet	switch-graph-7f184-7676f88dd9-z4frx	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503