Events

Time	Namespace	Component	RelatedObject	Reason	Message
12:11:31	kserve-ci-e2e-test		success-200-isvc-ec777-predictor-5c69cb57dc-l448d	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-ec777-predictor-5c69cb57dc-l448d to ip-10-0-140-177.ec2.internal
12:02:16	kserve-ci-e2e-test		success-200-isvc-e7782-predictor-74fd9b479-lhdr2	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-e7782-predictor-74fd9b479-lhdr2 to ip-10-0-140-177.ec2.internal
12:03:06	kserve-ci-e2e-test		error-404-isvc-b491e-predictor-7cd6c6456b-g4snd	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-b491e-predictor-7cd6c6456b-g4snd to ip-10-0-138-101.ec2.internal
12:12:42	kserve-ci-e2e-test		success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh to ip-10-0-140-177.ec2.internal
12:01:06	kserve-ci-e2e-test		success-200-isvc-462dc-predictor-8d4845d7-rtcpm	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-462dc-predictor-8d4845d7-rtcpm to ip-10-0-140-177.ec2.internal
12:02:17	kserve-ci-e2e-test		error-404-isvc-e7782-predictor-7fc45887b6-hdjss	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-e7782-predictor-7fc45887b6-hdjss to ip-10-0-138-101.ec2.internal
12:11:31	kserve-ci-e2e-test		error-404-isvc-ec777-predictor-6b58bbd7b6-cb5wt	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-ec777-predictor-6b58bbd7b6-cb5wt to ip-10-0-138-101.ec2.internal
11:50:21	kserve-ci-e2e-test		error-404-isvc-7cc25-predictor-f8874b7c6-5r6t2	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-7cc25-predictor-f8874b7c6-5r6t2 to ip-10-0-138-101.ec2.internal
11:52:41	kserve-ci-e2e-test		success-200-isvc-07b43-predictor-7b997f5f8f-q59st	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-07b43-predictor-7b997f5f8f-q59st to ip-10-0-140-177.ec2.internal
12:12:31	kserve-ci-e2e-test		splitter-graph-ec777-577d754886-kfjm6	Scheduled	Successfully assigned kserve-ci-e2e-test/splitter-graph-ec777-577d754886-kfjm6 to ip-10-0-140-177.ec2.internal
12:12:21	kserve-ci-e2e-test		success-200-isvc-beb62-predictor-74847944ff-snqr2	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-beb62-predictor-74847944ff-snqr2 to ip-10-0-140-177.ec2.internal
12:03:16	kserve-ci-e2e-test		ensemble-graph-e7782-6c5f799595-97lbg	Scheduled	Successfully assigned kserve-ci-e2e-test/ensemble-graph-e7782-6c5f799595-97lbg to ip-10-0-140-177.ec2.internal
12:13:42	kserve-ci-e2e-test		splitter-graph-5d4cc-9fb6994f-tzflz	Scheduled	Successfully assigned kserve-ci-e2e-test/splitter-graph-5d4cc-9fb6994f-tzflz to ip-10-0-140-177.ec2.internal
11:50:21	kserve-ci-e2e-test		isvc-xgboost-graph-predictor-555fc69d56-f4vm2	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4vm2 to ip-10-0-140-177.ec2.internal
11:51:41	kserve-ci-e2e-test		switch-graph-7cc25-6c975c9dbd-6v5mr	Scheduled	Successfully assigned kserve-ci-e2e-test/switch-graph-7cc25-6c975c9dbd-6v5mr to ip-10-0-140-177.ec2.internal
11:52:51	kserve-ci-e2e-test		switch-graph-5e1d2-69cf5bd554-gfrts	Scheduled	Successfully assigned kserve-ci-e2e-test/switch-graph-5e1d2-69cf5bd554-gfrts to ip-10-0-140-177.ec2.internal
12:13:21	kserve-ci-e2e-test		switch-graph-beb62-5c7d9675f8-nbnkw	Scheduled	Successfully assigned kserve-ci-e2e-test/switch-graph-beb62-5c7d9675f8-nbnkw to ip-10-0-140-177.ec2.internal
11:52:31	kserve-ci-e2e-test		model-chainer-6d7dd4967-4kmd4	Scheduled	Successfully assigned kserve-ci-e2e-test/model-chainer-6d7dd4967-4kmd4 to ip-10-0-140-177.ec2.internal
12:01:56	kserve-ci-e2e-test		error-404-isvc-a4899-predictor-7c8d7697d6-w47mt	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-a4899-predictor-7c8d7697d6-w47mt to ip-10-0-138-101.ec2.internal
12:03:06	kserve-ci-e2e-test		success-200-isvc-b491e-predictor-7559fc655f-hscwb	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-b491e-predictor-7559fc655f-hscwb to ip-10-0-140-177.ec2.internal
12:01:56	kserve-ci-e2e-test		success-200-isvc-a4899-predictor-65f55d68f9-f47hd	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-a4899-predictor-65f55d68f9-f47hd to ip-10-0-140-177.ec2.internal
12:04:06	kserve-ci-e2e-test		sequence-graph-b491e-5fcb67d95d-bqhpq	Scheduled	Successfully assigned kserve-ci-e2e-test/sequence-graph-b491e-5fcb67d95d-bqhpq to ip-10-0-140-177.ec2.internal
12:02:56	kserve-ci-e2e-test		sequence-graph-a4899-7dd75cd587-46gmq	Scheduled	Successfully assigned kserve-ci-e2e-test/sequence-graph-a4899-7dd75cd587-46gmq to ip-10-0-140-177.ec2.internal
11:50:21	kserve-ci-e2e-test		isvc-sklearn-graph-1-predictor-8b777f59f-2h82x	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-8b777f59f-2h82x to ip-10-0-140-177.ec2.internal
11:52:41	kserve-ci-e2e-test		error-404-isvc-07b43-predictor-755cf86f9b-hzktj	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-07b43-predictor-755cf86f9b-hzktj to ip-10-0-138-101.ec2.internal
12:01:06	kserve-ci-e2e-test		error-404-isvc-462dc-predictor-7cbccc4d4b-dlt97	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-462dc-predictor-7cbccc4d4b-dlt97 to ip-10-0-138-101.ec2.internal
11:51:51	kserve-ci-e2e-test		error-404-isvc-5e1d2-predictor-5cd9566866-r6bmq	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-5e1d2-predictor-5cd9566866-r6bmq to ip-10-0-138-101.ec2.internal
11:53:41	kserve-ci-e2e-test		sequence-graph-07b43-dccbfccf8-fw476	Scheduled	Successfully assigned kserve-ci-e2e-test/sequence-graph-07b43-dccbfccf8-fw476 to ip-10-0-140-177.ec2.internal
12:12:42	kserve-ci-e2e-test		error-404-isvc-5d4cc-predictor-59b9c7746b-g2bjw	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-5d4cc-predictor-59b9c7746b-g2bjw to ip-10-0-138-101.ec2.internal
11:51:51	kserve-ci-e2e-test		success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn to ip-10-0-140-177.ec2.internal
12:12:21	kserve-ci-e2e-test		error-404-isvc-beb62-predictor-64dc4bbbb-759qg	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-beb62-predictor-64dc4bbbb-759qg to ip-10-0-138-101.ec2.internal
11:50:21	kserve-ci-e2e-test		success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l to ip-10-0-140-177.ec2.internal
11:50:21	kserve-ci-e2e-test		isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp to ip-10-0-140-177.ec2.internal
12:02:06	kserve-ci-e2e-test		ensemble-graph-462dc-867dff89-xr95l	Scheduled	Successfully assigned kserve-ci-e2e-test/ensemble-graph-462dc-867dff89-xr95l to ip-10-0-140-177.ec2.internal
11:50:21	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-7cc25-predictor-57b4f6559c	SuccessfulCreate	Created pod: success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l
11:50:21	kserve-ci-e2e-test	deployment-controller	success-200-isvc-7cc25-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-7cc25-predictor-57b4f6559c from 0 to 1
11:50:21	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-graph-1-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-graph-1-predictor-8b777f59f from 0 to 1
11:50:21	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-graph-1-predictor-8b777f59f	SuccessfulCreate	Created pod: isvc-sklearn-graph-1-predictor-8b777f59f-2h82x
11:50:21	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
11:50:21	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-graph-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-graph-predictor-555fc69d56 from 0 to 1
11:50:21	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-graph-predictor-555fc69d56	SuccessfulCreate	Created pod: isvc-xgboost-graph-predictor-555fc69d56-f4vm2
11:50:21	kserve-ci-e2e-test	kubelet	success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l	Pulling	Pulling image "quay.io/opendatahub/success-200-isvc:odh-pr-1698"
11:50:21	kserve-ci-e2e-test	multus	success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l	AddedInterface	Add eth0 [10.134.0.18/23] from ovn-kubernetes
11:50:21	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-7cc25-predictor-f8874b7c6	SuccessfulCreate	Created pod: error-404-isvc-7cc25-predictor-f8874b7c6-5r6t2
11:50:21	kserve-ci-e2e-test	kubelet	error-404-isvc-7cc25-predictor-f8874b7c6-5r6t2	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "error-404-isvc-7cc25-predictor-serving-cert" not found
11:50:21	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-graph-2-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-graph-2-predictor-6c85f75dc7 from 0 to 1
11:50:21	kserve-ci-e2e-test	deployment-controller	error-404-isvc-7cc25-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-7cc25-predictor-f8874b7c6 from 0 to 1
11:50:21	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-graph-2-predictor-6c85f75dc7	SuccessfulCreate	Created pod: isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp
11:50:22	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4vm2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4e745fe12220230a0d4cc05dc90033c71b44df647b53b2249ee8bc2e75c04f9d" already present on machine
11:50:22	kserve-ci-e2e-test	multus	isvc-xgboost-graph-predictor-555fc69d56-f4vm2	AddedInterface	Add eth0 [10.134.0.20/23] from ovn-kubernetes
11:50:22 (x2)	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-8b777f59f-2h82x	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
11:50:22	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4vm2	Created	Created container: storage-initializer
11:50:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4e745fe12220230a0d4cc05dc90033c71b44df647b53b2249ee8bc2e75c04f9d" already present on machine
11:50:22	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4vm2	Started	Started container storage-initializer
11:50:22	kserve-ci-e2e-test	multus	error-404-isvc-7cc25-predictor-f8874b7c6-5r6t2	AddedInterface	Add eth0 [10.133.0.32/23] from ovn-kubernetes
11:50:22	kserve-ci-e2e-test	kubelet	error-404-isvc-7cc25-predictor-f8874b7c6-5r6t2	Pulling	Pulling image "quay.io/opendatahub/error-404-isvc:odh-pr-1698"
11:50:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp	Created	Created container: storage-initializer
11:50:22	kserve-ci-e2e-test	multus	isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp	AddedInterface	Add eth0 [10.134.0.21/23] from ovn-kubernetes
11:50:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-8b777f59f-2h82x	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4e745fe12220230a0d4cc05dc90033c71b44df647b53b2249ee8bc2e75c04f9d" already present on machine
11:50:23	kserve-ci-e2e-test	multus	isvc-sklearn-graph-1-predictor-8b777f59f-2h82x	AddedInterface	Add eth0 [10.134.0.19/23] from ovn-kubernetes
11:50:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-8b777f59f-2h82x	Created	Created container: storage-initializer
11:50:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp	Started	Started container storage-initializer
11:50:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-8b777f59f-2h82x	Started	Started container storage-initializer
11:50:27	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1698"
11:50:27	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4vm2	Pulling	Pulling image "kserve/xgbserver:latest"
11:50:31	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-8b777f59f-2h82x	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1698"
11:50:33	kserve-ci-e2e-test	kubelet	error-404-isvc-7cc25-predictor-f8874b7c6-5r6t2	Pulled	Successfully pulled image "quay.io/opendatahub/error-404-isvc:odh-pr-1698" in 11.26s (11.26s including waiting). Image size: 1334629465 bytes.
11:50:33	kserve-ci-e2e-test	kubelet	error-404-isvc-7cc25-predictor-f8874b7c6-5r6t2	Started	Started container kserve-container
11:50:33	kserve-ci-e2e-test	kubelet	error-404-isvc-7cc25-predictor-f8874b7c6-5r6t2	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
11:50:33	kserve-ci-e2e-test	kubelet	error-404-isvc-7cc25-predictor-f8874b7c6-5r6t2	Created	Created container: kserve-container
11:50:36	kserve-ci-e2e-test	kubelet	error-404-isvc-7cc25-predictor-f8874b7c6-5r6t2	Started	Started container kube-rbac-proxy
11:50:36	kserve-ci-e2e-test	kubelet	error-404-isvc-7cc25-predictor-f8874b7c6-5r6t2	Created	Created container: kube-rbac-proxy
11:50:36	kserve-ci-e2e-test	kubelet	error-404-isvc-7cc25-predictor-f8874b7c6-5r6t2	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.316s (2.316s including waiting). Image size: 211946088 bytes.
11:50:37	kserve-ci-e2e-test	kubelet	success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l	Created	Created container: kserve-container
11:50:37	kserve-ci-e2e-test	kubelet	success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l	Started	Started container kserve-container
11:50:37	kserve-ci-e2e-test	kubelet	success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
11:50:37	kserve-ci-e2e-test	kubelet	success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l	Pulled	Successfully pulled image "quay.io/opendatahub/success-200-isvc:odh-pr-1698" in 15.673s (15.673s including waiting). Image size: 1335616092 bytes.
11:50:37	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1698" in 9.582s (9.582s including waiting). Image size: 1555847076 bytes.
11:50:39	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-8b777f59f-2h82x	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1698" in 7.305s (7.305s including waiting). Image size: 1555847076 bytes.
11:50:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-8b777f59f-2h82x	Created	Created container: kube-rbac-proxy
11:50:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp	Started	Started container kserve-container
11:50:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-8b777f59f-2h82x	Started	Started container kserve-container
11:50:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-8b777f59f-2h82x	Created	Created container: kserve-container
11:50:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp	Created	Created container: kube-rbac-proxy
11:50:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 471ms (471ms including waiting). Image size: 211946088 bytes.
11:50:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
11:50:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-8b777f59f-2h82x	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 584ms (584ms including waiting). Image size: 211946088 bytes.
11:50:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp	Created	Created container: kserve-container
11:50:41	kserve-ci-e2e-test	kubelet	success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l	Started	Started container kube-rbac-proxy
11:50:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp	Started	Started container kube-rbac-proxy
11:50:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-8b777f59f-2h82x	Started	Started container kube-rbac-proxy
11:50:41	kserve-ci-e2e-test	kubelet	success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l	Created	Created container: kube-rbac-proxy
11:50:41	kserve-ci-e2e-test	kubelet	success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 3.956s (3.956s including waiting). Image size: 211946088 bytes.
11:50:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-8b777f59f-2h82x	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
11:50:51 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-7cc25-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:50:51 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-7cc25-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:50:55	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4vm2	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 27.652s (27.652s including waiting). Image size: 1305293051 bytes.
11:50:55	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4vm2	Created	Created container: kserve-container
11:50:55	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4vm2	Started	Started container kserve-container
11:50:55	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4vm2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
11:50:55	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4vm2	Created	Created container: kube-rbac-proxy
11:50:55	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4vm2	Started	Started container kube-rbac-proxy
11:51:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-7cc25-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:51:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-7cc25-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:51:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:51:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:51:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:51:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:51:18 (x6)	kserve-ci-e2e-test	kubelet	success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l	Unhealthy	Readiness probe failed: dial tcp 10.134.0.18:8080: connect: connection refused
11:51:21 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:51:21 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:51:22 (x5)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-7cc25	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
11:51:22	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-7cc25	InferenceServiceReady	InferenceService [error-404-isvc-7cc25] is Ready
11:51:28 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-7cc25	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
11:51:28	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-7cc25	InferenceServiceReady	InferenceService [success-200-isvc-7cc25] is Ready
11:51:41	kserve-ci-e2e-test	InferenceGraphController	switch-graph-7cc25	InferenceGraphReady	InferenceGraph [switch-graph-7cc25] is Ready
11:51:41	kserve-ci-e2e-test	kubelet	switch-graph-7cc25-6c975c9dbd-6v5mr	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "switch-graph-7cc25-serving-cert" not found
11:51:41	kserve-ci-e2e-test	deployment-controller	switch-graph-7cc25	ScalingReplicaSet	Scaled up replica set switch-graph-7cc25-6c975c9dbd from 0 to 1
11:51:41	kserve-ci-e2e-test	replicaset-controller	switch-graph-7cc25-6c975c9dbd	SuccessfulCreate	Created pod: switch-graph-7cc25-6c975c9dbd-6v5mr
11:51:42	kserve-ci-e2e-test	multus	switch-graph-7cc25-6c975c9dbd-6v5mr	AddedInterface	Add eth0 [10.134.0.22/23] from ovn-kubernetes
11:51:42	kserve-ci-e2e-test	kubelet	switch-graph-7cc25-6c975c9dbd-6v5mr	Pulling	Pulling image "quay.io/opendatahub/kserve-router@sha256:3ecf25ca4988bb9c678b6995d3cd17dde00648bf8dca69fe7b78fe19fd5c9b14"
11:51:44	kserve-ci-e2e-test	kubelet	switch-graph-7cc25-6c975c9dbd-6v5mr	Started	Started container switch-graph-7cc25
11:51:44	kserve-ci-e2e-test	kubelet	switch-graph-7cc25-6c975c9dbd-6v5mr	Created	Created container: switch-graph-7cc25
11:51:44	kserve-ci-e2e-test	kubelet	switch-graph-7cc25-6c975c9dbd-6v5mr	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:3ecf25ca4988bb9c678b6995d3cd17dde00648bf8dca69fe7b78fe19fd5c9b14" in 2.369s (2.369s including waiting). Image size: 218072664 bytes.
11:51:51	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-5e1d2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-5e1d2": the object has been modified; please apply your changes to the latest version and try again
11:51:51	kserve-ci-e2e-test	kubelet	success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
11:51:51	kserve-ci-e2e-test	kubelet	error-404-isvc-7cc25-predictor-f8874b7c6-5r6t2	Killing	Stopping container kube-rbac-proxy
11:51:51	kserve-ci-e2e-test	kubelet	switch-graph-7cc25-6c975c9dbd-6v5mr	Killing	Stopping container switch-graph-7cc25
11:51:51 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-7cc25-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:51:51	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-5e1d2	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-5e1d2": the object has been modified; please apply your changes to the latest version and try again
11:51:51	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-5e1d2	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-5e1d2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-5e1d2": the object has been modified; please apply your changes to the latest version and try again
11:51:51 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-7cc25-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:51:51 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-7cc25-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:51:51 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-7cc25-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:51:51	kserve-ci-e2e-test	kubelet	success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l	Killing	Stopping container kube-rbac-proxy
11:51:51	kserve-ci-e2e-test	kubelet	success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l	Killing	Stopping container kserve-container
11:51:51	kserve-ci-e2e-test	deployment-controller	error-404-isvc-5e1d2-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-5e1d2-predictor-5cd9566866 from 0 to 1
11:51:51	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-5e1d2-predictor-5cd9566866	SuccessfulCreate	Created pod: error-404-isvc-5e1d2-predictor-5cd9566866-r6bmq
11:51:51	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-5e1d2	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-5e1d2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-5e1d2": the object has been modified; please apply your changes to the latest version and try again
11:51:51	kserve-ci-e2e-test	deployment-controller	success-200-isvc-5e1d2-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-5e1d2-predictor-78746df7ff from 0 to 1
11:51:51	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-5e1d2-predictor-78746df7ff	SuccessfulCreate	Created pod: success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn
11:51:51	kserve-ci-e2e-test	kubelet	error-404-isvc-7cc25-predictor-f8874b7c6-5r6t2	Killing	Stopping container kserve-container
11:51:51	kserve-ci-e2e-test	kubelet	success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn	Started	Started container kserve-container
11:51:51	kserve-ci-e2e-test	kubelet	success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn	Created	Created container: kserve-container
11:51:51	kserve-ci-e2e-test	kubelet	success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1698" already present on machine
11:51:51	kserve-ci-e2e-test	multus	success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn	AddedInterface	Add eth0 [10.134.0.23/23] from ovn-kubernetes
11:51:52	kserve-ci-e2e-test	kubelet	error-404-isvc-5e1d2-predictor-5cd9566866-r6bmq	Started	Started container kube-rbac-proxy
11:51:52	kserve-ci-e2e-test	kubelet	error-404-isvc-5e1d2-predictor-5cd9566866-r6bmq	Created	Created container: kserve-container
11:51:52	kserve-ci-e2e-test	kubelet	success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn	Created	Created container: kube-rbac-proxy
11:51:52	kserve-ci-e2e-test	kubelet	error-404-isvc-5e1d2-predictor-5cd9566866-r6bmq	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
11:51:52 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-7cc25-predictor-f8874b7c6-5r6t2	Unhealthy	Readiness probe failed: dial tcp 10.133.0.32:8080: connect: connection refused
11:51:52	kserve-ci-e2e-test	kubelet	error-404-isvc-5e1d2-predictor-5cd9566866-r6bmq	Started	Started container kserve-container
11:51:52	kserve-ci-e2e-test	multus	error-404-isvc-5e1d2-predictor-5cd9566866-r6bmq	AddedInterface	Add eth0 [10.133.0.33/23] from ovn-kubernetes
11:51:52	kserve-ci-e2e-test	kubelet	error-404-isvc-7cc25-predictor-f8874b7c6-5r6t2	Unhealthy	Readiness probe failed: Get "https://10.133.0.32:8643/healthz": dial tcp 10.133.0.32:8643: connect: connection refused
11:51:52	kserve-ci-e2e-test	kubelet	error-404-isvc-5e1d2-predictor-5cd9566866-r6bmq	Created	Created container: kube-rbac-proxy
11:51:52	kserve-ci-e2e-test	kubelet	success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn	Started	Started container kube-rbac-proxy
11:51:52	kserve-ci-e2e-test	kubelet	error-404-isvc-5e1d2-predictor-5cd9566866-r6bmq	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1698" already present on machine
11:51:53	kserve-ci-e2e-test	kubelet	success-200-isvc-7cc25-predictor-57b4f6559c-p6z7l	Unhealthy	Readiness probe failed: Get "https://10.134.0.18:8643/healthz": dial tcp 10.134.0.18:8643: connect: connection refused
11:51:57 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-8b777f59f-2h82x	Unhealthy	Readiness probe failed: dial tcp 10.134.0.19:8080: connect: connection refused
11:52:02	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-graph	InferenceServiceReady	InferenceService [isvc-xgboost-graph] is Ready
11:52:02 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-graph	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
11:52:04	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-2	InferenceServiceReady	InferenceService [isvc-sklearn-graph-2] is Ready
11:52:04 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
11:52:07 (x10)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
11:52:07	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-1	InferenceServiceReady	InferenceService [isvc-sklearn-graph-1] is Ready
11:52:21 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-5e1d2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:52:21 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-5e1d2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:52:21 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:52:21 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-5e1d2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:52:21 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-5e1d2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:52:21 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:52:21 (x6)	kserve-ci-e2e-test	kubelet	switch-graph-7cc25-6c975c9dbd-6v5mr	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
11:52:31	kserve-ci-e2e-test	InferenceGraphController	model-chainer	InferenceGraphReady	InferenceGraph [model-chainer] is Ready
11:52:31	kserve-ci-e2e-test	deployment-controller	model-chainer	ScalingReplicaSet	Scaled up replica set model-chainer-6d7dd4967 from 0 to 1
11:52:31	kserve-ci-e2e-test	replicaset-controller	model-chainer-6d7dd4967	SuccessfulCreate	Created pod: model-chainer-6d7dd4967-4kmd4
11:52:31	kserve-ci-e2e-test	kubelet	model-chainer-6d7dd4967-4kmd4	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-serving-cert" not found
11:52:32	kserve-ci-e2e-test	kubelet	model-chainer-6d7dd4967-4kmd4	Started	Started container model-chainer
11:52:32	kserve-ci-e2e-test	multus	model-chainer-6d7dd4967-4kmd4	AddedInterface	Add eth0 [10.134.0.24/23] from ovn-kubernetes
11:52:32	kserve-ci-e2e-test	kubelet	model-chainer-6d7dd4967-4kmd4	Created	Created container: model-chainer
11:52:32	kserve-ci-e2e-test	kubelet	model-chainer-6d7dd4967-4kmd4	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:3ecf25ca4988bb9c678b6995d3cd17dde00648bf8dca69fe7b78fe19fd5c9b14" already present on machine
11:52:36 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:52:36 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:52:36 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:52:36 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:52:39 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-5e1d2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
11:52:39	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-5e1d2	InferenceServiceReady	InferenceService [success-200-isvc-5e1d2] is Ready
11:52:39	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-5e1d2	InferenceServiceReady	InferenceService [error-404-isvc-5e1d2] is Ready
11:52:39 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-5e1d2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
11:52:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4vm2	Unhealthy	Readiness probe failed: Get "https://10.134.0.20:8643/healthz": dial tcp 10.134.0.20:8643: connect: connection refused
11:52:41	kserve-ci-e2e-test	deployment-controller	error-404-isvc-07b43-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-07b43-predictor-755cf86f9b from 0 to 1
11:52:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4vm2	Killing	Stopping container kube-rbac-proxy
11:52:41	kserve-ci-e2e-test	deployment-controller	success-200-isvc-07b43-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-07b43-predictor-7b997f5f8f from 0 to 1
11:52:41	kserve-ci-e2e-test	kubelet	error-404-isvc-07b43-predictor-755cf86f9b-hzktj	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "error-404-isvc-07b43-predictor-serving-cert" not found
11:52:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-8b777f59f-2h82x	Killing	Stopping container kserve-container
11:52:41 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4vm2	Unhealthy	Readiness probe failed: dial tcp 10.134.0.20:8080: connect: connection refused
11:52:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp	Killing	Stopping container kube-rbac-proxy
11:52:41	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-07b43-predictor-755cf86f9b	SuccessfulCreate	Created pod: error-404-isvc-07b43-predictor-755cf86f9b-hzktj
11:52:41	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4vm2	Killing	Stopping container kserve-container
11:52:41	kserve-ci-e2e-test	kubelet	model-chainer-6d7dd4967-4kmd4	Killing	Stopping container model-chainer
11:52:41	kserve-ci-e2e-test	multus	success-200-isvc-07b43-predictor-7b997f5f8f-q59st	AddedInterface	Add eth0 [10.134.0.25/23] from ovn-kubernetes
11:52:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp	Killing	Stopping container kserve-container
11:52:41	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-07b43	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-07b43": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-07b43": the object has been modified; please apply your changes to the latest version and try again
11:52:41	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-07b43	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-07b43": the object has been modified; please apply your changes to the latest version and try again
11:52:41	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-8b777f59f-2h82x	Killing	Stopping container kube-rbac-proxy
11:52:41	kserve-ci-e2e-test	kubelet	success-200-isvc-07b43-predictor-7b997f5f8f-q59st	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1698" already present on machine
11:52:41	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-07b43-predictor-7b997f5f8f	SuccessfulCreate	Created pod: success-200-isvc-07b43-predictor-7b997f5f8f-q59st
11:52:41	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-07b43	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-07b43": the object has been modified; please apply your changes to the latest version and try again
11:52:41	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-07b43	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-07b43": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-07b43": the object has been modified; please apply your changes to the latest version and try again
11:52:42	kserve-ci-e2e-test	kubelet	error-404-isvc-07b43-predictor-755cf86f9b-hzktj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
11:52:42	kserve-ci-e2e-test	kubelet	error-404-isvc-07b43-predictor-755cf86f9b-hzktj	Started	Started container kube-rbac-proxy
11:52:42	kserve-ci-e2e-test	multus	error-404-isvc-07b43-predictor-755cf86f9b-hzktj	AddedInterface	Add eth0 [10.133.0.34/23] from ovn-kubernetes
11:52:42	kserve-ci-e2e-test	kubelet	error-404-isvc-07b43-predictor-755cf86f9b-hzktj	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1698" already present on machine
11:52:42	kserve-ci-e2e-test	kubelet	success-200-isvc-07b43-predictor-7b997f5f8f-q59st	Started	Started container kube-rbac-proxy
11:52:42	kserve-ci-e2e-test	kubelet	success-200-isvc-07b43-predictor-7b997f5f8f-q59st	Created	Created container: kube-rbac-proxy
11:52:42	kserve-ci-e2e-test	kubelet	success-200-isvc-07b43-predictor-7b997f5f8f-q59st	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
11:52:42	kserve-ci-e2e-test	kubelet	success-200-isvc-07b43-predictor-7b997f5f8f-q59st	Started	Started container kserve-container
11:52:42	kserve-ci-e2e-test	kubelet	success-200-isvc-07b43-predictor-7b997f5f8f-q59st	Created	Created container: kserve-container
11:52:42	kserve-ci-e2e-test	kubelet	error-404-isvc-07b43-predictor-755cf86f9b-hzktj	Created	Created container: kserve-container
11:52:42	kserve-ci-e2e-test	kubelet	error-404-isvc-07b43-predictor-755cf86f9b-hzktj	Started	Started container kserve-container
11:52:42	kserve-ci-e2e-test	kubelet	error-404-isvc-07b43-predictor-755cf86f9b-hzktj	Created	Created container: kube-rbac-proxy
11:52:42	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp	Unhealthy	Readiness probe failed: Get "https://10.134.0.21:8643/healthz": dial tcp 10.134.0.21:8643: connect: connection refused
11:52:43	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-8b777f59f-2h82x	Unhealthy	Readiness probe failed: Get "https://10.134.0.19:8643/healthz": dial tcp 10.134.0.19:8643: connect: connection refused
11:52:44 (x11)	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-6c85f75dc7-z8mrp	Unhealthy	Readiness probe failed: dial tcp 10.134.0.21:8080: connect: connection refused
11:52:51	kserve-ci-e2e-test	kubelet	switch-graph-5e1d2-69cf5bd554-gfrts	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "switch-graph-5e1d2-serving-cert" not found
11:52:51	kserve-ci-e2e-test	replicaset-controller	switch-graph-5e1d2-69cf5bd554	SuccessfulCreate	Created pod: switch-graph-5e1d2-69cf5bd554-gfrts
11:52:51	kserve-ci-e2e-test	deployment-controller	switch-graph-5e1d2	ScalingReplicaSet	Scaled up replica set switch-graph-5e1d2-69cf5bd554 from 0 to 1
11:52:51	kserve-ci-e2e-test	InferenceGraphController	switch-graph-5e1d2	InferenceGraphReady	InferenceGraph [switch-graph-5e1d2] is Ready
11:52:52	kserve-ci-e2e-test	kubelet	switch-graph-5e1d2-69cf5bd554-gfrts	Started	Started container switch-graph-5e1d2
11:52:52	kserve-ci-e2e-test	kubelet	switch-graph-5e1d2-69cf5bd554-gfrts	Created	Created container: switch-graph-5e1d2
11:52:52	kserve-ci-e2e-test	multus	switch-graph-5e1d2-69cf5bd554-gfrts	AddedInterface	Add eth0 [10.134.0.26/23] from ovn-kubernetes
11:52:52	kserve-ci-e2e-test	kubelet	switch-graph-5e1d2-69cf5bd554-gfrts	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:3ecf25ca4988bb9c678b6995d3cd17dde00648bf8dca69fe7b78fe19fd5c9b14" already present on machine
11:53:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-5e1d2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:53:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-5e1d2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:53:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-5e1d2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:53:06 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-5e1d2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:53:09 (x6)	kserve-ci-e2e-test	kubelet	model-chainer-6d7dd4967-4kmd4	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
11:53:11 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-07b43-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:53:11 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-07b43-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:53:21 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-5e1d2	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:53:21 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-5e1d2	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:53:26 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-07b43-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:53:26 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-07b43-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:53:28	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-07b43	InferenceServiceReady	InferenceService [success-200-isvc-07b43] is Ready
11:53:28 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-07b43	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
11:53:30	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-07b43	InferenceServiceReady	InferenceService [error-404-isvc-07b43] is Ready
11:53:30 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-07b43	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
11:53:41	kserve-ci-e2e-test	InferenceGraphController	sequence-graph-07b43	InferenceGraphReady	InferenceGraph [sequence-graph-07b43] is Ready
11:53:41	kserve-ci-e2e-test	kubelet	sequence-graph-07b43-dccbfccf8-fw476	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "sequence-graph-07b43-serving-cert" not found
11:53:41	kserve-ci-e2e-test	replicaset-controller	sequence-graph-07b43-dccbfccf8	SuccessfulCreate	Created pod: sequence-graph-07b43-dccbfccf8-fw476
11:53:41	kserve-ci-e2e-test	deployment-controller	sequence-graph-07b43	ScalingReplicaSet	Scaled up replica set sequence-graph-07b43-dccbfccf8 from 0 to 1
11:53:42	kserve-ci-e2e-test	multus	sequence-graph-07b43-dccbfccf8-fw476	AddedInterface	Add eth0 [10.134.0.27/23] from ovn-kubernetes
11:53:42	kserve-ci-e2e-test	kubelet	sequence-graph-07b43-dccbfccf8-fw476	Started	Started container sequence-graph-07b43
11:53:42	kserve-ci-e2e-test	kubelet	sequence-graph-07b43-dccbfccf8-fw476	Created	Created container: sequence-graph-07b43
11:53:42	kserve-ci-e2e-test	kubelet	sequence-graph-07b43-dccbfccf8-fw476	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:3ecf25ca4988bb9c678b6995d3cd17dde00648bf8dca69fe7b78fe19fd5c9b14" already present on machine
11:53:56	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-07b43	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:53:56	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-07b43	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
11:53:56 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-07b43-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:53:56 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-07b43-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:54:11	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-07b43	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:54:11	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-07b43	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:54:26 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-07b43-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
11:54:26 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-07b43-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:01:06	kserve-ci-e2e-test	kubelet	success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn	Killing	Stopping container kube-rbac-proxy
12:01:06	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-462dc	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-462dc": the object has been modified; please apply your changes to the latest version and try again
12:01:06	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-462dc-predictor-8d4845d7	SuccessfulCreate	Created pod: success-200-isvc-462dc-predictor-8d4845d7-rtcpm
12:01:06	kserve-ci-e2e-test	deployment-controller	success-200-isvc-462dc-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-462dc-predictor-8d4845d7 from 0 to 1
12:01:06	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-462dc	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-462dc": the object has been modified; please apply your changes to the latest version and try again
12:01:06	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-462dc	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-462dc": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-462dc": the object has been modified; please apply your changes to the latest version and try again
12:01:06	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-462dc	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-462dc": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-462dc": the object has been modified; please apply your changes to the latest version and try again
12:01:06	kserve-ci-e2e-test	deployment-controller	error-404-isvc-462dc-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-462dc-predictor-7cbccc4d4b from 0 to 1
12:01:06	kserve-ci-e2e-test	kubelet	switch-graph-5e1d2-69cf5bd554-gfrts	Killing	Stopping container switch-graph-5e1d2
12:01:06	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-462dc-predictor-7cbccc4d4b	SuccessfulCreate	Created pod: error-404-isvc-462dc-predictor-7cbccc4d4b-dlt97
12:01:06	kserve-ci-e2e-test	kubelet	error-404-isvc-5e1d2-predictor-5cd9566866-r6bmq	Killing	Stopping container kserve-container
12:01:06	kserve-ci-e2e-test	kubelet	error-404-isvc-5e1d2-predictor-5cd9566866-r6bmq	Killing	Stopping container kube-rbac-proxy
12:01:06	kserve-ci-e2e-test	kubelet	success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn	Killing	Stopping container kserve-container
12:01:06	kserve-ci-e2e-test	kubelet	success-200-isvc-462dc-predictor-8d4845d7-rtcpm	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "success-200-isvc-462dc-predictor-serving-cert" not found
12:01:07	kserve-ci-e2e-test	multus	error-404-isvc-462dc-predictor-7cbccc4d4b-dlt97	AddedInterface	Add eth0 [10.133.0.35/23] from ovn-kubernetes
12:01:07	kserve-ci-e2e-test	kubelet	error-404-isvc-462dc-predictor-7cbccc4d4b-dlt97	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1698" already present on machine
12:01:07	kserve-ci-e2e-test	kubelet	error-404-isvc-462dc-predictor-7cbccc4d4b-dlt97	Created	Created container: kserve-container
12:01:07	kserve-ci-e2e-test	kubelet	error-404-isvc-462dc-predictor-7cbccc4d4b-dlt97	Started	Started container kube-rbac-proxy
12:01:07	kserve-ci-e2e-test	kubelet	error-404-isvc-462dc-predictor-7cbccc4d4b-dlt97	Created	Created container: kube-rbac-proxy
12:01:07	kserve-ci-e2e-test	multus	success-200-isvc-462dc-predictor-8d4845d7-rtcpm	AddedInterface	Add eth0 [10.134.0.28/23] from ovn-kubernetes
12:01:07	kserve-ci-e2e-test	kubelet	success-200-isvc-462dc-predictor-8d4845d7-rtcpm	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1698" already present on machine
12:01:07	kserve-ci-e2e-test	kubelet	success-200-isvc-462dc-predictor-8d4845d7-rtcpm	Created	Created container: kserve-container
12:01:07	kserve-ci-e2e-test	kubelet	success-200-isvc-462dc-predictor-8d4845d7-rtcpm	Started	Started container kserve-container
12:01:07	kserve-ci-e2e-test	kubelet	error-404-isvc-462dc-predictor-7cbccc4d4b-dlt97	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
12:01:07	kserve-ci-e2e-test	kubelet	success-200-isvc-462dc-predictor-8d4845d7-rtcpm	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
12:01:07	kserve-ci-e2e-test	kubelet	success-200-isvc-462dc-predictor-8d4845d7-rtcpm	Created	Created container: kube-rbac-proxy
12:01:07	kserve-ci-e2e-test	kubelet	success-200-isvc-462dc-predictor-8d4845d7-rtcpm	Started	Started container kube-rbac-proxy
12:01:07	kserve-ci-e2e-test	kubelet	error-404-isvc-462dc-predictor-7cbccc4d4b-dlt97	Started	Started container kserve-container
12:01:09	kserve-ci-e2e-test	kubelet	error-404-isvc-5e1d2-predictor-5cd9566866-r6bmq	Unhealthy	Readiness probe failed: Get "https://10.133.0.33:8643/healthz": dial tcp 10.133.0.33:8643: connect: connection refused
12:01:09	kserve-ci-e2e-test	kubelet	success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn	Unhealthy	Readiness probe failed: Get "https://10.134.0.23:8643/healthz": dial tcp 10.134.0.23:8643: connect: connection refused
12:01:09 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-5e1d2-predictor-5cd9566866-r6bmq	Unhealthy	Readiness probe failed: dial tcp 10.133.0.33:8080: connect: connection refused
12:01:09 (x7)	kserve-ci-e2e-test	kubelet	success-200-isvc-5e1d2-predictor-78746df7ff-7lkrn	Unhealthy	Readiness probe failed: dial tcp 10.134.0.23:8080: connect: connection refused
12:01:34 (x6)	kserve-ci-e2e-test	kubelet	switch-graph-5e1d2-69cf5bd554-gfrts	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
12:01:43 (x6)	kserve-ci-e2e-test	kubelet	error-404-isvc-462dc-predictor-7cbccc4d4b-dlt97	Unhealthy	Readiness probe failed: dial tcp 10.133.0.35:8080: connect: connection refused
12:01:44 (x6)	kserve-ci-e2e-test	kubelet	success-200-isvc-462dc-predictor-8d4845d7-rtcpm	Unhealthy	Readiness probe failed: dial tcp 10.134.0.28:8080: connect: connection refused
12:01:51 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-462dc-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:01:51 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-462dc-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:01:51 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-462dc-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:01:51 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-462dc-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:01:53	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-462dc	InferenceServiceReady	InferenceService [error-404-isvc-462dc] is Ready
12:01:53 (x5)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-462dc	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
12:01:54 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-462dc	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
12:01:54	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-462dc	InferenceServiceReady	InferenceService [success-200-isvc-462dc] is Ready
12:01:56	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-a4899	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-a4899": the object has been modified; please apply your changes to the latest version and try again
12:01:56	kserve-ci-e2e-test	kubelet	error-404-isvc-a4899-predictor-7c8d7697d6-w47mt	Created	Created container: kserve-container
12:01:56	kserve-ci-e2e-test	kubelet	error-404-isvc-a4899-predictor-7c8d7697d6-w47mt	Started	Started container kube-rbac-proxy
12:01:56	kserve-ci-e2e-test	kubelet	error-404-isvc-a4899-predictor-7c8d7697d6-w47mt	Created	Created container: kube-rbac-proxy
12:01:56	kserve-ci-e2e-test	kubelet	error-404-isvc-07b43-predictor-755cf86f9b-hzktj	Killing	Stopping container kserve-container
12:01:56	kserve-ci-e2e-test	kubelet	success-200-isvc-07b43-predictor-7b997f5f8f-q59st	Killing	Stopping container kserve-container
12:01:56	kserve-ci-e2e-test	kubelet	success-200-isvc-07b43-predictor-7b997f5f8f-q59st	Killing	Stopping container kube-rbac-proxy
12:01:56	kserve-ci-e2e-test	kubelet	error-404-isvc-07b43-predictor-755cf86f9b-hzktj	Killing	Stopping container kube-rbac-proxy
12:01:56	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-a4899	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-a4899": the object has been modified; please apply your changes to the latest version and try again
12:01:56	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-a4899	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-a4899": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-a4899": the object has been modified; please apply your changes to the latest version and try again
12:01:56	kserve-ci-e2e-test	deployment-controller	success-200-isvc-a4899-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-a4899-predictor-65f55d68f9 from 0 to 1
12:01:56	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-a4899-predictor-65f55d68f9	SuccessfulCreate	Created pod: success-200-isvc-a4899-predictor-65f55d68f9-f47hd
12:01:56	kserve-ci-e2e-test	kubelet	success-200-isvc-a4899-predictor-65f55d68f9-f47hd	Started	Started container kube-rbac-proxy
12:01:56	kserve-ci-e2e-test	kubelet	error-404-isvc-a4899-predictor-7c8d7697d6-w47mt	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
12:01:56	kserve-ci-e2e-test	kubelet	error-404-isvc-a4899-predictor-7c8d7697d6-w47mt	Started	Started container kserve-container
12:01:56	kserve-ci-e2e-test	multus	success-200-isvc-a4899-predictor-65f55d68f9-f47hd	AddedInterface	Add eth0 [10.134.0.29/23] from ovn-kubernetes
12:01:56	kserve-ci-e2e-test	kubelet	error-404-isvc-a4899-predictor-7c8d7697d6-w47mt	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1698" already present on machine
12:01:56	kserve-ci-e2e-test	multus	error-404-isvc-a4899-predictor-7c8d7697d6-w47mt	AddedInterface	Add eth0 [10.133.0.36/23] from ovn-kubernetes
12:01:56	kserve-ci-e2e-test	kubelet	success-200-isvc-a4899-predictor-65f55d68f9-f47hd	Created	Created container: kube-rbac-proxy
12:01:56	kserve-ci-e2e-test	kubelet	success-200-isvc-a4899-predictor-65f55d68f9-f47hd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
12:01:56	kserve-ci-e2e-test	kubelet	success-200-isvc-a4899-predictor-65f55d68f9-f47hd	Started	Started container kserve-container
12:01:56	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-a4899	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-a4899": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-a4899": the object has been modified; please apply your changes to the latest version and try again
12:01:56	kserve-ci-e2e-test	kubelet	success-200-isvc-a4899-predictor-65f55d68f9-f47hd	Created	Created container: kserve-container
12:01:56	kserve-ci-e2e-test	kubelet	success-200-isvc-a4899-predictor-65f55d68f9-f47hd	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1698" already present on machine
12:01:56	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-a4899-predictor-7c8d7697d6	SuccessfulCreate	Created pod: error-404-isvc-a4899-predictor-7c8d7697d6-w47mt
12:01:56	kserve-ci-e2e-test	deployment-controller	error-404-isvc-a4899-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-a4899-predictor-7c8d7697d6 from 0 to 1
12:01:56	kserve-ci-e2e-test	kubelet	sequence-graph-07b43-dccbfccf8-fw476	Killing	Stopping container sequence-graph-07b43
12:01:58	kserve-ci-e2e-test	kubelet	success-200-isvc-07b43-predictor-7b997f5f8f-q59st	Unhealthy	Readiness probe failed: Get "https://10.134.0.25:8643/healthz": dial tcp 10.134.0.25:8643: connect: connection refused
12:01:58 (x7)	kserve-ci-e2e-test	kubelet	success-200-isvc-07b43-predictor-7b997f5f8f-q59st	Unhealthy	Readiness probe failed: dial tcp 10.134.0.25:8080: connect: connection refused
12:02:00	kserve-ci-e2e-test	kubelet	error-404-isvc-07b43-predictor-755cf86f9b-hzktj	Unhealthy	Readiness probe failed: Get "https://10.133.0.34:8643/healthz": dial tcp 10.133.0.34:8643: connect: connection refused
12:02:00 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-07b43-predictor-755cf86f9b-hzktj	Unhealthy	Readiness probe failed: dial tcp 10.133.0.34:8080: connect: connection refused
12:02:06	kserve-ci-e2e-test	replicaset-controller	ensemble-graph-462dc-867dff89	SuccessfulCreate	Created pod: ensemble-graph-462dc-867dff89-xr95l
12:02:06	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-462dc-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:02:06	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-462dc-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:02:06	kserve-ci-e2e-test	InferenceGraphController	ensemble-graph-462dc	InferenceGraphReady	InferenceGraph [ensemble-graph-462dc] is Ready
12:02:06	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-462dc-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:02:06	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-462dc-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:02:06	kserve-ci-e2e-test	deployment-controller	ensemble-graph-462dc	ScalingReplicaSet	Scaled up replica set ensemble-graph-462dc-867dff89 from 0 to 1
12:02:06	kserve-ci-e2e-test	kubelet	ensemble-graph-462dc-867dff89-xr95l	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "ensemble-graph-462dc-serving-cert" not found
12:02:07	kserve-ci-e2e-test	kubelet	ensemble-graph-462dc-867dff89-xr95l	Created	Created container: ensemble-graph-462dc
12:02:07	kserve-ci-e2e-test	multus	ensemble-graph-462dc-867dff89-xr95l	AddedInterface	Add eth0 [10.134.0.30/23] from ovn-kubernetes
12:02:07	kserve-ci-e2e-test	kubelet	ensemble-graph-462dc-867dff89-xr95l	Started	Started container ensemble-graph-462dc
12:02:07	kserve-ci-e2e-test	kubelet	ensemble-graph-462dc-867dff89-xr95l	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:3ecf25ca4988bb9c678b6995d3cd17dde00648bf8dca69fe7b78fe19fd5c9b14" already present on machine
12:02:11	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-a4899-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:02:11	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-a4899-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:02:16	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-e7782	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-e7782": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-e7782": the object has been modified; please apply your changes to the latest version and try again
12:02:16	kserve-ci-e2e-test	deployment-controller	success-200-isvc-e7782-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-e7782-predictor-74fd9b479 from 0 to 1
12:02:16	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-e7782	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-e7782": the object has been modified; please apply your changes to the latest version and try again
12:02:16	kserve-ci-e2e-test	kubelet	ensemble-graph-462dc-867dff89-xr95l	Killing	Stopping container ensemble-graph-462dc
12:02:16	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-e7782-predictor-74fd9b479	SuccessfulCreate	Created pod: success-200-isvc-e7782-predictor-74fd9b479-lhdr2
12:02:16	kserve-ci-e2e-test	kubelet	error-404-isvc-462dc-predictor-7cbccc4d4b-dlt97	Killing	Stopping container kube-rbac-proxy
12:02:16	kserve-ci-e2e-test	kubelet	error-404-isvc-462dc-predictor-7cbccc4d4b-dlt97	Killing	Stopping container kserve-container
12:02:17	kserve-ci-e2e-test	kubelet	error-404-isvc-e7782-predictor-7fc45887b6-hdjss	Created	Created container: kserve-container
12:02:17	kserve-ci-e2e-test	kubelet	success-200-isvc-e7782-predictor-74fd9b479-lhdr2	Created	Created container: kube-rbac-proxy
12:02:17	kserve-ci-e2e-test	kubelet	error-404-isvc-e7782-predictor-7fc45887b6-hdjss	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
12:02:17	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-e7782	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-e7782": the object has been modified; please apply your changes to the latest version and try again
12:02:17	kserve-ci-e2e-test	kubelet	error-404-isvc-e7782-predictor-7fc45887b6-hdjss	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1698" already present on machine
12:02:17	kserve-ci-e2e-test	multus	error-404-isvc-e7782-predictor-7fc45887b6-hdjss	AddedInterface	Add eth0 [10.133.0.37/23] from ovn-kubernetes
12:02:17	kserve-ci-e2e-test	kubelet	error-404-isvc-e7782-predictor-7fc45887b6-hdjss	Created	Created container: kube-rbac-proxy
12:02:17	kserve-ci-e2e-test	kubelet	error-404-isvc-e7782-predictor-7fc45887b6-hdjss	Started	Started container kube-rbac-proxy
12:02:17	kserve-ci-e2e-test	kubelet	success-200-isvc-e7782-predictor-74fd9b479-lhdr2	Started	Started container kube-rbac-proxy
12:02:17	kserve-ci-e2e-test	kubelet	error-404-isvc-e7782-predictor-7fc45887b6-hdjss	Started	Started container kserve-container
12:02:17	kserve-ci-e2e-test	kubelet	success-200-isvc-e7782-predictor-74fd9b479-lhdr2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
12:02:17	kserve-ci-e2e-test	kubelet	success-200-isvc-e7782-predictor-74fd9b479-lhdr2	Started	Started container kserve-container
12:02:17	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-e7782-predictor-7fc45887b6	SuccessfulCreate	Created pod: error-404-isvc-e7782-predictor-7fc45887b6-hdjss
12:02:17	kserve-ci-e2e-test	kubelet	success-200-isvc-462dc-predictor-8d4845d7-rtcpm	Killing	Stopping container kserve-container
12:02:17	kserve-ci-e2e-test	deployment-controller	error-404-isvc-e7782-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-e7782-predictor-7fc45887b6 from 0 to 1
12:02:17	kserve-ci-e2e-test	kubelet	success-200-isvc-e7782-predictor-74fd9b479-lhdr2	Created	Created container: kserve-container
12:02:17	kserve-ci-e2e-test	kubelet	success-200-isvc-462dc-predictor-8d4845d7-rtcpm	Killing	Stopping container kube-rbac-proxy
12:02:17	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-e7782	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-e7782": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-e7782": the object has been modified; please apply your changes to the latest version and try again
12:02:17	kserve-ci-e2e-test	multus	success-200-isvc-e7782-predictor-74fd9b479-lhdr2	AddedInterface	Add eth0 [10.134.0.31/23] from ovn-kubernetes
12:02:17	kserve-ci-e2e-test	kubelet	success-200-isvc-e7782-predictor-74fd9b479-lhdr2	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1698" already present on machine
12:02:18	kserve-ci-e2e-test	kubelet	error-404-isvc-462dc-predictor-7cbccc4d4b-dlt97	Unhealthy	Readiness probe failed: Get "https://10.133.0.35:8643/healthz": dial tcp 10.133.0.35:8643: connect: connection refused
12:02:19	kserve-ci-e2e-test	kubelet	success-200-isvc-462dc-predictor-8d4845d7-rtcpm	Unhealthy	Readiness probe failed: Get "https://10.134.0.28:8643/healthz": dial tcp 10.134.0.28:8643: connect: connection refused
12:02:23 (x6)	kserve-ci-e2e-test	kubelet	sequence-graph-07b43-dccbfccf8-fw476	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
12:02:33 (x6)	kserve-ci-e2e-test	kubelet	success-200-isvc-a4899-predictor-65f55d68f9-f47hd	Unhealthy	Readiness probe failed: dial tcp 10.134.0.29:8080: connect: connection refused
12:02:34 (x6)	kserve-ci-e2e-test	kubelet	error-404-isvc-a4899-predictor-7c8d7697d6-w47mt	Unhealthy	Readiness probe failed: dial tcp 10.133.0.36:8080: connect: connection refused
12:02:41 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-a4899-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:02:41 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-a4899-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:02:43	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-a4899	InferenceServiceReady	InferenceService [success-200-isvc-a4899] is Ready
12:02:43 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-a4899	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
12:02:44 (x5)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-a4899	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
12:02:44 (x6)	kserve-ci-e2e-test	kubelet	ensemble-graph-462dc-867dff89-xr95l	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
12:02:44	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-a4899	InferenceServiceReady	InferenceService [error-404-isvc-a4899] is Ready
12:02:46	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-e7782-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:02:46	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-e7782-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:02:47 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-e7782-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:02:47 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-e7782-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:02:56	kserve-ci-e2e-test	InferenceGraphController	sequence-graph-a4899	InferenceGraphReady	InferenceGraph [sequence-graph-a4899] is Ready
12:02:56	kserve-ci-e2e-test	kubelet	sequence-graph-a4899-7dd75cd587-46gmq	Created	Created container: sequence-graph-a4899
12:02:56	kserve-ci-e2e-test	replicaset-controller	sequence-graph-a4899-7dd75cd587	SuccessfulCreate	Created pod: sequence-graph-a4899-7dd75cd587-46gmq
12:02:56	kserve-ci-e2e-test	deployment-controller	sequence-graph-a4899	ScalingReplicaSet	Scaled up replica set sequence-graph-a4899-7dd75cd587 from 0 to 1
12:02:56	kserve-ci-e2e-test	kubelet	sequence-graph-a4899-7dd75cd587-46gmq	Started	Started container sequence-graph-a4899
12:02:56	kserve-ci-e2e-test	kubelet	sequence-graph-a4899-7dd75cd587-46gmq	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:3ecf25ca4988bb9c678b6995d3cd17dde00648bf8dca69fe7b78fe19fd5c9b14" already present on machine
12:02:56	kserve-ci-e2e-test	multus	sequence-graph-a4899-7dd75cd587-46gmq	AddedInterface	Add eth0 [10.134.0.32/23] from ovn-kubernetes
12:02:56 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-a4899-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:02:56 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-a4899-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:02:56	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-a4899-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:02:56	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-a4899-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:03:03 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-e7782	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
12:03:03	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-e7782	InferenceServiceReady	InferenceService [success-200-isvc-e7782] is Ready
12:03:04 (x5)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-e7782	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
12:03:04	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-e7782	InferenceServiceReady	InferenceService [error-404-isvc-e7782] is Ready
12:03:06	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-b491e	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-b491e": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-b491e": the object has been modified; please apply your changes to the latest version and try again
12:03:06	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-b491e-predictor-7559fc655f	SuccessfulCreate	Created pod: success-200-isvc-b491e-predictor-7559fc655f-hscwb
12:03:06	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-b491e	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-b491e": the object has been modified; please apply your changes to the latest version and try again
12:03:06	kserve-ci-e2e-test	kubelet	error-404-isvc-a4899-predictor-7c8d7697d6-w47mt	Killing	Stopping container kserve-container
12:03:06	kserve-ci-e2e-test	kubelet	error-404-isvc-a4899-predictor-7c8d7697d6-w47mt	Killing	Stopping container kube-rbac-proxy
12:03:06	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-b491e	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-b491e": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-b491e": the object has been modified; please apply your changes to the latest version and try again
12:03:06	kserve-ci-e2e-test	deployment-controller	success-200-isvc-b491e-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-b491e-predictor-7559fc655f from 0 to 1
12:03:06	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-b491e	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-b491e": the object has been modified; please apply your changes to the latest version and try again
12:03:06	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-b491e-predictor-7cd6c6456b	SuccessfulCreate	Created pod: error-404-isvc-b491e-predictor-7cd6c6456b-g4snd
12:03:06	kserve-ci-e2e-test	kubelet	success-200-isvc-b491e-predictor-7559fc655f-hscwb	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "success-200-isvc-b491e-predictor-serving-cert" not found
12:03:06	kserve-ci-e2e-test	kubelet	sequence-graph-a4899-7dd75cd587-46gmq	Killing	Stopping container sequence-graph-a4899
12:03:06	kserve-ci-e2e-test	kubelet	success-200-isvc-a4899-predictor-65f55d68f9-f47hd	Killing	Stopping container kserve-container
12:03:06	kserve-ci-e2e-test	kubelet	success-200-isvc-a4899-predictor-65f55d68f9-f47hd	Killing	Stopping container kube-rbac-proxy
12:03:06	kserve-ci-e2e-test	deployment-controller	error-404-isvc-b491e-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-b491e-predictor-7cd6c6456b from 0 to 1
12:03:07	kserve-ci-e2e-test	kubelet	error-404-isvc-b491e-predictor-7cd6c6456b-g4snd	Created	Created container: kserve-container
12:03:07	kserve-ci-e2e-test	kubelet	success-200-isvc-b491e-predictor-7559fc655f-hscwb	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1698" already present on machine
12:03:07	kserve-ci-e2e-test	kubelet	success-200-isvc-b491e-predictor-7559fc655f-hscwb	Created	Created container: kube-rbac-proxy
12:03:07	kserve-ci-e2e-test	kubelet	success-200-isvc-b491e-predictor-7559fc655f-hscwb	Started	Started container kube-rbac-proxy
12:03:07	kserve-ci-e2e-test	kubelet	error-404-isvc-b491e-predictor-7cd6c6456b-g4snd	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1698" already present on machine
12:03:07	kserve-ci-e2e-test	kubelet	error-404-isvc-b491e-predictor-7cd6c6456b-g4snd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
12:03:07	kserve-ci-e2e-test	kubelet	success-200-isvc-b491e-predictor-7559fc655f-hscwb	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
12:03:07	kserve-ci-e2e-test	kubelet	error-404-isvc-b491e-predictor-7cd6c6456b-g4snd	Started	Started container kube-rbac-proxy
12:03:07	kserve-ci-e2e-test	kubelet	success-200-isvc-b491e-predictor-7559fc655f-hscwb	Started	Started container kserve-container
12:03:07	kserve-ci-e2e-test	kubelet	success-200-isvc-b491e-predictor-7559fc655f-hscwb	Created	Created container: kserve-container
12:03:07	kserve-ci-e2e-test	multus	error-404-isvc-b491e-predictor-7cd6c6456b-g4snd	AddedInterface	Add eth0 [10.133.0.38/23] from ovn-kubernetes
12:03:07	kserve-ci-e2e-test	multus	success-200-isvc-b491e-predictor-7559fc655f-hscwb	AddedInterface	Add eth0 [10.134.0.33/23] from ovn-kubernetes
12:03:07	kserve-ci-e2e-test	kubelet	error-404-isvc-b491e-predictor-7cd6c6456b-g4snd	Started	Started container kserve-container
12:03:07	kserve-ci-e2e-test	kubelet	error-404-isvc-b491e-predictor-7cd6c6456b-g4snd	Created	Created container: kube-rbac-proxy
12:03:08	kserve-ci-e2e-test	kubelet	success-200-isvc-a4899-predictor-65f55d68f9-f47hd	Unhealthy	Readiness probe failed: Get "https://10.134.0.29:8643/healthz": dial tcp 10.134.0.29:8643: connect: connection refused
12:03:10	kserve-ci-e2e-test	kubelet	error-404-isvc-a4899-predictor-7c8d7697d6-w47mt	Unhealthy	Readiness probe failed: Get "https://10.133.0.36:8643/healthz": context deadline exceeded
12:03:16	kserve-ci-e2e-test	InferenceGraphController	ensemble-graph-e7782	InferenceGraphReady	InferenceGraph [ensemble-graph-e7782] is Ready
12:03:16	kserve-ci-e2e-test	replicaset-controller	ensemble-graph-e7782-6c5f799595	SuccessfulCreate	Created pod: ensemble-graph-e7782-6c5f799595-97lbg
12:03:16	kserve-ci-e2e-test	deployment-controller	ensemble-graph-e7782	ScalingReplicaSet	Scaled up replica set ensemble-graph-e7782-6c5f799595 from 0 to 1
12:03:17	kserve-ci-e2e-test	multus	ensemble-graph-e7782-6c5f799595-97lbg	AddedInterface	Add eth0 [10.134.0.34/23] from ovn-kubernetes
12:03:17 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-e7782-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:03:17	kserve-ci-e2e-test	kubelet	ensemble-graph-e7782-6c5f799595-97lbg	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "ensemble-graph-e7782-serving-cert" not found
12:03:17	kserve-ci-e2e-test	kubelet	ensemble-graph-e7782-6c5f799595-97lbg	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:3ecf25ca4988bb9c678b6995d3cd17dde00648bf8dca69fe7b78fe19fd5c9b14" already present on machine
12:03:17	kserve-ci-e2e-test	kubelet	ensemble-graph-e7782-6c5f799595-97lbg	Created	Created container: ensemble-graph-e7782
12:03:17	kserve-ci-e2e-test	kubelet	ensemble-graph-e7782-6c5f799595-97lbg	Started	Started container ensemble-graph-e7782
12:03:17 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-e7782-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:03:31 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-e7782-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:03:31 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-e7782-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:03:33 (x6)	kserve-ci-e2e-test	kubelet	sequence-graph-a4899-7dd75cd587-46gmq	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
12:03:36 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-b491e-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:03:36 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-b491e-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:03:36 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-b491e-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:03:36 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-b491e-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:03:44 (x6)	kserve-ci-e2e-test	kubelet	success-200-isvc-b491e-predictor-7559fc655f-hscwb	Unhealthy	Readiness probe failed: dial tcp 10.134.0.33:8080: connect: connection refused
12:03:46 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	ensemble-graph-e7782	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:03:46 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	ensemble-graph-e7782	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:03:54 (x5)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-b491e	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
12:03:54	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-b491e	InferenceServiceReady	InferenceService [success-200-isvc-b491e] is Ready
12:03:54	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-b491e	InferenceServiceReady	InferenceService [error-404-isvc-b491e] is Ready
12:03:54 (x6)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-b491e	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
12:04:06	kserve-ci-e2e-test	replicaset-controller	sequence-graph-b491e-5fcb67d95d	SuccessfulCreate	Created pod: sequence-graph-b491e-5fcb67d95d-bqhpq
12:04:06	kserve-ci-e2e-test	InferenceGraphController	sequence-graph-b491e	InferenceGraphReady	InferenceGraph [sequence-graph-b491e] is Ready
12:04:06	kserve-ci-e2e-test	deployment-controller	sequence-graph-b491e	ScalingReplicaSet	Scaled up replica set sequence-graph-b491e-5fcb67d95d from 0 to 1
12:04:06	kserve-ci-e2e-test	kubelet	sequence-graph-b491e-5fcb67d95d-bqhpq	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "sequence-graph-b491e-serving-cert" not found
12:04:07	kserve-ci-e2e-test	kubelet	sequence-graph-b491e-5fcb67d95d-bqhpq	Started	Started container sequence-graph-b491e
12:04:07	kserve-ci-e2e-test	kubelet	sequence-graph-b491e-5fcb67d95d-bqhpq	Created	Created container: sequence-graph-b491e
12:04:07	kserve-ci-e2e-test	kubelet	sequence-graph-b491e-5fcb67d95d-bqhpq	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:3ecf25ca4988bb9c678b6995d3cd17dde00648bf8dca69fe7b78fe19fd5c9b14" already present on machine
12:04:07	kserve-ci-e2e-test	multus	sequence-graph-b491e-5fcb67d95d-bqhpq	AddedInterface	Add eth0 [10.134.0.35/23] from ovn-kubernetes
12:04:21 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-b491e-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:04:21 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-b491e-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:04:36 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-b491e-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:04:36 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-b491e	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:04:36 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-b491e-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:04:36 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-b491e	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:11:31	kserve-ci-e2e-test	deployment-controller	success-200-isvc-ec777-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-ec777-predictor-5c69cb57dc from 0 to 1
12:11:31	kserve-ci-e2e-test	deployment-controller	error-404-isvc-ec777-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-ec777-predictor-6b58bbd7b6 from 0 to 1
12:11:31	kserve-ci-e2e-test	kubelet	success-200-isvc-e7782-predictor-74fd9b479-lhdr2	Killing	Stopping container kube-rbac-proxy
12:11:31	kserve-ci-e2e-test	kubelet	success-200-isvc-e7782-predictor-74fd9b479-lhdr2	Killing	Stopping container kserve-container
12:11:31	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-ec777	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-ec777": the object has been modified; please apply your changes to the latest version and try again
12:11:31	kserve-ci-e2e-test	kubelet	ensemble-graph-e7782-6c5f799595-97lbg	Killing	Stopping container ensemble-graph-e7782
12:11:31	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-ec777	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-ec777": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-ec777": the object has been modified; please apply your changes to the latest version and try again
12:11:31	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-ec777-predictor-6b58bbd7b6	SuccessfulCreate	Created pod: error-404-isvc-ec777-predictor-6b58bbd7b6-cb5wt
12:11:31	kserve-ci-e2e-test	kubelet	success-200-isvc-ec777-predictor-5c69cb57dc-l448d	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "success-200-isvc-ec777-predictor-serving-cert" not found
12:11:31	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-ec777-predictor-5c69cb57dc	SuccessfulCreate	Created pod: success-200-isvc-ec777-predictor-5c69cb57dc-l448d
12:11:32	kserve-ci-e2e-test	kubelet	success-200-isvc-ec777-predictor-5c69cb57dc-l448d	Started	Started container kserve-container
12:11:32	kserve-ci-e2e-test	multus	success-200-isvc-ec777-predictor-5c69cb57dc-l448d	AddedInterface	Add eth0 [10.134.0.36/23] from ovn-kubernetes
12:11:32	kserve-ci-e2e-test	kubelet	success-200-isvc-ec777-predictor-5c69cb57dc-l448d	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
12:11:32	kserve-ci-e2e-test	kubelet	success-200-isvc-ec777-predictor-5c69cb57dc-l448d	Created	Created container: kube-rbac-proxy
12:11:32	kserve-ci-e2e-test	kubelet	error-404-isvc-e7782-predictor-7fc45887b6-hdjss	Killing	Stopping container kserve-container
12:11:32	kserve-ci-e2e-test	kubelet	error-404-isvc-ec777-predictor-6b58bbd7b6-cb5wt	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "error-404-isvc-ec777-predictor-serving-cert" not found
12:11:32	kserve-ci-e2e-test	kubelet	success-200-isvc-ec777-predictor-5c69cb57dc-l448d	Started	Started container kube-rbac-proxy
12:11:32	kserve-ci-e2e-test	kubelet	success-200-isvc-ec777-predictor-5c69cb57dc-l448d	Created	Created container: kserve-container
12:11:32	kserve-ci-e2e-test	kubelet	success-200-isvc-ec777-predictor-5c69cb57dc-l448d	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1698" already present on machine
12:11:32	kserve-ci-e2e-test	multus	error-404-isvc-ec777-predictor-6b58bbd7b6-cb5wt	AddedInterface	Add eth0 [10.133.0.39/23] from ovn-kubernetes
12:11:32	kserve-ci-e2e-test	kubelet	error-404-isvc-e7782-predictor-7fc45887b6-hdjss	Killing	Stopping container kube-rbac-proxy
12:11:33	kserve-ci-e2e-test	kubelet	error-404-isvc-ec777-predictor-6b58bbd7b6-cb5wt	Started	Started container kube-rbac-proxy
12:11:33	kserve-ci-e2e-test	kubelet	error-404-isvc-ec777-predictor-6b58bbd7b6-cb5wt	Started	Started container kserve-container
12:11:33 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-e7782-predictor-7fc45887b6-hdjss	Unhealthy	Readiness probe failed: dial tcp 10.133.0.37:8080: connect: connection refused
12:11:33 (x7)	kserve-ci-e2e-test	kubelet	success-200-isvc-e7782-predictor-74fd9b479-lhdr2	Unhealthy	Readiness probe failed: dial tcp 10.134.0.31:8080: connect: connection refused
12:11:33	kserve-ci-e2e-test	kubelet	error-404-isvc-ec777-predictor-6b58bbd7b6-cb5wt	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
12:11:33	kserve-ci-e2e-test	kubelet	error-404-isvc-ec777-predictor-6b58bbd7b6-cb5wt	Created	Created container: kube-rbac-proxy
12:11:33	kserve-ci-e2e-test	kubelet	success-200-isvc-e7782-predictor-74fd9b479-lhdr2	Unhealthy	Readiness probe failed: Get "https://10.134.0.31:8643/healthz": dial tcp 10.134.0.31:8643: connect: connection refused
12:11:33	kserve-ci-e2e-test	kubelet	error-404-isvc-ec777-predictor-6b58bbd7b6-cb5wt	Created	Created container: kserve-container
12:11:33	kserve-ci-e2e-test	kubelet	error-404-isvc-ec777-predictor-6b58bbd7b6-cb5wt	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1698" already present on machine
12:11:33	kserve-ci-e2e-test	kubelet	error-404-isvc-e7782-predictor-7fc45887b6-hdjss	Unhealthy	Readiness probe failed: Get "https://10.133.0.37:8643/healthz": dial tcp 10.133.0.37:8643: connect: connection refused
12:11:59 (x6)	kserve-ci-e2e-test	kubelet	ensemble-graph-e7782-6c5f799595-97lbg	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
12:12:01	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-ec777-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:12:01	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-ec777-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:12:01 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-ec777-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:12:01 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-ec777-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:12:09 (x6)	kserve-ci-e2e-test	kubelet	success-200-isvc-ec777-predictor-5c69cb57dc-l448d	Unhealthy	Readiness probe failed: dial tcp 10.134.0.36:8080: connect: connection refused
12:12:10 (x6)	kserve-ci-e2e-test	kubelet	error-404-isvc-ec777-predictor-6b58bbd7b6-cb5wt	Unhealthy	Readiness probe failed: dial tcp 10.133.0.39:8080: connect: connection refused
12:12:19 (x6)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-ec777	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
12:12:19	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-ec777	InferenceServiceReady	InferenceService [success-200-isvc-ec777] is Ready
12:12:20	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-ec777	InferenceServiceReady	InferenceService [error-404-isvc-ec777] is Ready
12:12:20 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-ec777	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
12:12:21	kserve-ci-e2e-test	kubelet	success-200-isvc-b491e-predictor-7559fc655f-hscwb	Killing	Stopping container kserve-container
12:12:21	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-beb62	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-beb62": the object has been modified; please apply your changes to the latest version and try again
12:12:21	kserve-ci-e2e-test	kubelet	error-404-isvc-b491e-predictor-7cd6c6456b-g4snd	Killing	Stopping container kube-rbac-proxy
12:12:21	kserve-ci-e2e-test	kubelet	error-404-isvc-b491e-predictor-7cd6c6456b-g4snd	Killing	Stopping container kserve-container
12:12:21	kserve-ci-e2e-test	kubelet	error-404-isvc-beb62-predictor-64dc4bbbb-759qg	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "error-404-isvc-beb62-predictor-serving-cert" not found
12:12:21	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-beb62	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-beb62": the object has been modified; please apply your changes to the latest version and try again
12:12:21	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-beb62	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-beb62": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-beb62": the object has been modified; please apply your changes to the latest version and try again
12:12:21	kserve-ci-e2e-test	deployment-controller	success-200-isvc-beb62-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-beb62-predictor-74847944ff from 0 to 1
12:12:21	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-beb62-predictor-74847944ff	SuccessfulCreate	Created pod: success-200-isvc-beb62-predictor-74847944ff-snqr2
12:12:21	kserve-ci-e2e-test	kubelet	success-200-isvc-beb62-predictor-74847944ff-snqr2	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "success-200-isvc-beb62-predictor-serving-cert" not found
12:12:21	kserve-ci-e2e-test	kubelet	success-200-isvc-b491e-predictor-7559fc655f-hscwb	Killing	Stopping container kube-rbac-proxy
12:12:21	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-beb62-predictor-64dc4bbbb	SuccessfulCreate	Created pod: error-404-isvc-beb62-predictor-64dc4bbbb-759qg
12:12:21	kserve-ci-e2e-test	deployment-controller	error-404-isvc-beb62-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-beb62-predictor-64dc4bbbb from 0 to 1
12:12:21	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-beb62	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-beb62": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-beb62": the object has been modified; please apply your changes to the latest version and try again
12:12:21	kserve-ci-e2e-test	kubelet	sequence-graph-b491e-5fcb67d95d-bqhpq	Killing	Stopping container sequence-graph-b491e
12:12:22	kserve-ci-e2e-test	multus	success-200-isvc-beb62-predictor-74847944ff-snqr2	AddedInterface	Add eth0 [10.134.0.37/23] from ovn-kubernetes
12:12:22	kserve-ci-e2e-test	kubelet	error-404-isvc-beb62-predictor-64dc4bbbb-759qg	Created	Created container: kserve-container
12:12:22	kserve-ci-e2e-test	multus	error-404-isvc-beb62-predictor-64dc4bbbb-759qg	AddedInterface	Add eth0 [10.133.0.40/23] from ovn-kubernetes
12:12:22	kserve-ci-e2e-test	kubelet	success-200-isvc-beb62-predictor-74847944ff-snqr2	Created	Created container: kserve-container
12:12:22	kserve-ci-e2e-test	kubelet	error-404-isvc-beb62-predictor-64dc4bbbb-759qg	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1698" already present on machine
12:12:22	kserve-ci-e2e-test	kubelet	success-200-isvc-beb62-predictor-74847944ff-snqr2	Started	Started container kserve-container
12:12:22	kserve-ci-e2e-test	kubelet	error-404-isvc-beb62-predictor-64dc4bbbb-759qg	Started	Started container kube-rbac-proxy
12:12:22	kserve-ci-e2e-test	kubelet	success-200-isvc-beb62-predictor-74847944ff-snqr2	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1698" already present on machine
12:12:22	kserve-ci-e2e-test	kubelet	error-404-isvc-beb62-predictor-64dc4bbbb-759qg	Started	Started container kserve-container
12:12:22	kserve-ci-e2e-test	kubelet	error-404-isvc-beb62-predictor-64dc4bbbb-759qg	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
12:12:22	kserve-ci-e2e-test	kubelet	error-404-isvc-beb62-predictor-64dc4bbbb-759qg	Created	Created container: kube-rbac-proxy
12:12:22	kserve-ci-e2e-test	kubelet	success-200-isvc-beb62-predictor-74847944ff-snqr2	Started	Started container kube-rbac-proxy
12:12:22	kserve-ci-e2e-test	kubelet	success-200-isvc-beb62-predictor-74847944ff-snqr2	Created	Created container: kube-rbac-proxy
12:12:22	kserve-ci-e2e-test	kubelet	success-200-isvc-beb62-predictor-74847944ff-snqr2	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
12:12:24 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-b491e-predictor-7cd6c6456b-g4snd	Unhealthy	Readiness probe failed: dial tcp 10.133.0.38:8080: connect: connection refused
12:12:24	kserve-ci-e2e-test	kubelet	error-404-isvc-b491e-predictor-7cd6c6456b-g4snd	Unhealthy	Readiness probe failed: Get "https://10.133.0.38:8643/healthz": dial tcp 10.133.0.38:8643: connect: connection refused
12:12:31 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-ec777-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:12:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-ec777-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:12:31	kserve-ci-e2e-test	replicaset-controller	splitter-graph-ec777-577d754886	SuccessfulCreate	Created pod: splitter-graph-ec777-577d754886-kfjm6
12:12:31	kserve-ci-e2e-test	deployment-controller	splitter-graph-ec777	ScalingReplicaSet	Scaled up replica set splitter-graph-ec777-577d754886 from 0 to 1
12:12:31	kserve-ci-e2e-test	InferenceGraphController	splitter-graph-ec777	InferenceGraphReady	InferenceGraph [splitter-graph-ec777] is Ready
12:12:31 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-ec777-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:12:31 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-ec777-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:12:32	kserve-ci-e2e-test	kubelet	splitter-graph-ec777-577d754886-kfjm6	Started	Started container splitter-graph-ec777
12:12:32	kserve-ci-e2e-test	kubelet	splitter-graph-ec777-577d754886-kfjm6	Created	Created container: splitter-graph-ec777
12:12:32	kserve-ci-e2e-test	kubelet	splitter-graph-ec777-577d754886-kfjm6	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:3ecf25ca4988bb9c678b6995d3cd17dde00648bf8dca69fe7b78fe19fd5c9b14" already present on machine
12:12:32	kserve-ci-e2e-test	multus	splitter-graph-ec777-577d754886-kfjm6	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
12:12:36	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-beb62-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:12:36	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-beb62-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:12:41	kserve-ci-e2e-test	kubelet	splitter-graph-ec777-577d754886-kfjm6	Killing	Stopping container splitter-graph-ec777
12:12:41	kserve-ci-e2e-test	kubelet	success-200-isvc-ec777-predictor-5c69cb57dc-l448d	Killing	Stopping container kserve-container
12:12:41	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-5d4cc	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-5d4cc": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-5d4cc": the object has been modified; please apply your changes to the latest version and try again
12:12:41	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-5d4cc	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-5d4cc": the object has been modified; please apply your changes to the latest version and try again
12:12:41	kserve-ci-e2e-test	deployment-controller	success-200-isvc-5d4cc-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-5d4cc-predictor-745d8d8b55 from 0 to 1
12:12:41	kserve-ci-e2e-test	kubelet	success-200-isvc-ec777-predictor-5c69cb57dc-l448d	Killing	Stopping container kube-rbac-proxy
12:12:42	kserve-ci-e2e-test	kubelet	error-404-isvc-ec777-predictor-6b58bbd7b6-cb5wt	Killing	Stopping container kube-rbac-proxy
12:12:42	kserve-ci-e2e-test	kubelet	error-404-isvc-5d4cc-predictor-59b9c7746b-g2bjw	Created	Created container: kserve-container
12:12:42	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-5d4cc-predictor-745d8d8b55	SuccessfulCreate	Created pod: success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh
12:12:42	kserve-ci-e2e-test	multus	error-404-isvc-5d4cc-predictor-59b9c7746b-g2bjw	AddedInterface	Add eth0 [10.133.0.41/23] from ovn-kubernetes
12:12:42	kserve-ci-e2e-test	kubelet	error-404-isvc-5d4cc-predictor-59b9c7746b-g2bjw	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1698" already present on machine
12:12:42	kserve-ci-e2e-test	deployment-controller	error-404-isvc-5d4cc-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-5d4cc-predictor-59b9c7746b from 0 to 1
12:12:42	kserve-ci-e2e-test	kubelet	error-404-isvc-5d4cc-predictor-59b9c7746b-g2bjw	Started	Started container kserve-container
12:12:42	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-5d4cc	UpdateFailed	Failed to update status for InferenceService "error-404-isvc-5d4cc": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-5d4cc": the object has been modified; please apply your changes to the latest version and try again
12:12:42	kserve-ci-e2e-test	kubelet	error-404-isvc-5d4cc-predictor-59b9c7746b-g2bjw	Started	Started container kube-rbac-proxy
12:12:42	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-5d4cc-predictor-59b9c7746b	SuccessfulCreate	Created pod: error-404-isvc-5d4cc-predictor-59b9c7746b-g2bjw
12:12:42	kserve-ci-e2e-test	kubelet	error-404-isvc-ec777-predictor-6b58bbd7b6-cb5wt	Killing	Stopping container kserve-container
12:12:42	kserve-ci-e2e-test	kubelet	success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "success-200-isvc-5d4cc-predictor-serving-cert" not found
12:12:42	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-5d4cc	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-5d4cc": the object has been modified; please apply your changes to the latest version and try again
12:12:42	kserve-ci-e2e-test	kubelet	error-404-isvc-5d4cc-predictor-59b9c7746b-g2bjw	Created	Created container: kube-rbac-proxy
12:12:42	kserve-ci-e2e-test	kubelet	error-404-isvc-5d4cc-predictor-59b9c7746b-g2bjw	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
12:12:43	kserve-ci-e2e-test	kubelet	success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1698" already present on machine
12:12:43	kserve-ci-e2e-test	multus	success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
12:12:43	kserve-ci-e2e-test	kubelet	success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh	Created	Created container: kserve-container
12:12:43	kserve-ci-e2e-test	kubelet	success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh	Started	Started container kserve-container
12:12:43	kserve-ci-e2e-test	kubelet	success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
12:12:43	kserve-ci-e2e-test	kubelet	success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh	Created	Created container: kube-rbac-proxy
12:12:43	kserve-ci-e2e-test	kubelet	success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh	Started	Started container kube-rbac-proxy
12:12:44	kserve-ci-e2e-test	kubelet	success-200-isvc-ec777-predictor-5c69cb57dc-l448d	Unhealthy	Readiness probe failed: Get "https://10.134.0.36:8643/healthz": dial tcp 10.134.0.36:8643: connect: connection refused
12:12:45	kserve-ci-e2e-test	kubelet	error-404-isvc-ec777-predictor-6b58bbd7b6-cb5wt	Unhealthy	Readiness probe failed: Get "https://10.133.0.39:8643/healthz": dial tcp 10.133.0.39:8643: connect: connection refused
12:12:49 (x6)	kserve-ci-e2e-test	kubelet	sequence-graph-b491e-5fcb67d95d-bqhpq	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
12:12:51 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-beb62-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:12:51 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-beb62-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:12:59 (x6)	kserve-ci-e2e-test	kubelet	success-200-isvc-beb62-predictor-74847944ff-snqr2	Unhealthy	Readiness probe failed: dial tcp 10.134.0.37:8080: connect: connection refused
12:12:59 (x6)	kserve-ci-e2e-test	kubelet	error-404-isvc-beb62-predictor-64dc4bbbb-759qg	Unhealthy	Readiness probe failed: dial tcp 10.133.0.40:8080: connect: connection refused
12:13:08 (x6)	kserve-ci-e2e-test	kubelet	splitter-graph-ec777-577d754886-kfjm6	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
12:13:09	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-beb62	InferenceServiceReady	InferenceService [error-404-isvc-beb62] is Ready
12:13:09 (x5)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-beb62	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
12:13:10	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-beb62	InferenceServiceReady	InferenceService [success-200-isvc-beb62] is Ready
12:13:10 (x6)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-beb62	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
12:13:21	kserve-ci-e2e-test	deployment-controller	switch-graph-beb62	ScalingReplicaSet	Scaled up replica set switch-graph-beb62-5c7d9675f8 from 0 to 1
12:13:21	kserve-ci-e2e-test	InferenceGraphController	switch-graph-beb62	InferenceGraphReady	InferenceGraph [switch-graph-beb62] is Ready
12:13:21 (x6)	kserve-ci-e2e-test	kubelet	success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh	Unhealthy	Readiness probe failed: dial tcp 10.134.0.39:8080: connect: connection refused
12:13:21	kserve-ci-e2e-test	replicaset-controller	switch-graph-beb62-5c7d9675f8	SuccessfulCreate	Created pod: switch-graph-beb62-5c7d9675f8-nbnkw
12:13:22 (x2)	kserve-ci-e2e-test	kubelet	switch-graph-beb62-5c7d9675f8-nbnkw	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "switch-graph-beb62-serving-cert" not found
12:13:23	kserve-ci-e2e-test	multus	switch-graph-beb62-5c7d9675f8-nbnkw	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
12:13:23	kserve-ci-e2e-test	kubelet	switch-graph-beb62-5c7d9675f8-nbnkw	Started	Started container switch-graph-beb62
12:13:23	kserve-ci-e2e-test	kubelet	switch-graph-beb62-5c7d9675f8-nbnkw	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:3ecf25ca4988bb9c678b6995d3cd17dde00648bf8dca69fe7b78fe19fd5c9b14" already present on machine
12:13:23	kserve-ci-e2e-test	kubelet	switch-graph-beb62-5c7d9675f8-nbnkw	Created	Created container: switch-graph-beb62
12:13:27 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-5d4cc-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:13:27 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-5d4cc-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:13:27 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-5d4cc-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:13:27 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-5d4cc-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:13:29	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-5d4cc	InferenceServiceReady	InferenceService [error-404-isvc-5d4cc] is Ready
12:13:29 (x5)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-5d4cc	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
12:13:31 (x6)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-5d4cc	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
12:13:31	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-5d4cc	InferenceServiceReady	InferenceService [success-200-isvc-5d4cc] is Ready
12:13:36 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-beb62-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:13:36 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-beb62-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:13:36 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-beb62-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:13:36 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-beb62-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:13:42	kserve-ci-e2e-test	deployment-controller	splitter-graph-5d4cc	ScalingReplicaSet	Scaled up replica set splitter-graph-5d4cc-9fb6994f from 0 to 1
12:13:42	kserve-ci-e2e-test	replicaset-controller	splitter-graph-5d4cc-9fb6994f	SuccessfulCreate	Created pod: splitter-graph-5d4cc-9fb6994f-tzflz
12:13:42	kserve-ci-e2e-test	multus	splitter-graph-5d4cc-9fb6994f-tzflz	AddedInterface	Add eth0 [10.134.0.41/23] from ovn-kubernetes
12:13:42	kserve-ci-e2e-test	kubelet	splitter-graph-5d4cc-9fb6994f-tzflz	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:3ecf25ca4988bb9c678b6995d3cd17dde00648bf8dca69fe7b78fe19fd5c9b14" already present on machine
12:13:42	kserve-ci-e2e-test	kubelet	splitter-graph-5d4cc-9fb6994f-tzflz	Created	Created container: splitter-graph-5d4cc
12:13:42	kserve-ci-e2e-test	kubelet	splitter-graph-5d4cc-9fb6994f-tzflz	Started	Started container splitter-graph-5d4cc
12:13:42	kserve-ci-e2e-test	InferenceGraphController	splitter-graph-5d4cc	InferenceGraphReady	InferenceGraph [splitter-graph-5d4cc] is Ready
12:13:51 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-beb62	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:13:51 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-beb62	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:13:57 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-5d4cc-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:13:57 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-5d4cc-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:13:57 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-5d4cc-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:13:57 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-5d4cc-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
12:14:27 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	splitter-graph-5d4cc	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:14:27 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	splitter-graph-5d4cc	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
12:21:56	kserve-ci-e2e-test	kubelet	success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh	Killing	Stopping container kserve-container
12:21:56	kserve-ci-e2e-test	kubelet	success-200-isvc-5d4cc-predictor-745d8d8b55-bscnh	Killing	Stopping container kube-rbac-proxy
12:21:56	kserve-ci-e2e-test	kubelet	error-404-isvc-5d4cc-predictor-59b9c7746b-g2bjw	Killing	Stopping container kube-rbac-proxy
12:21:56	kserve-ci-e2e-test	kubelet	splitter-graph-5d4cc-9fb6994f-tzflz	Killing	Stopping container splitter-graph-5d4cc
12:21:56	kserve-ci-e2e-test	kubelet	error-404-isvc-5d4cc-predictor-59b9c7746b-g2bjw	Killing	Stopping container kserve-container
12:21:59 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-5d4cc-predictor-59b9c7746b-g2bjw	Unhealthy	Readiness probe failed: dial tcp 10.133.0.41:8080: connect: connection refused
12:21:59	kserve-ci-e2e-test	kubelet	error-404-isvc-5d4cc-predictor-59b9c7746b-g2bjw	Unhealthy	Readiness probe failed: Get "https://10.133.0.41:8643/healthz": dial tcp 10.133.0.41:8643: connect: connection refused
12:22:24 (x6)	kserve-ci-e2e-test	kubelet	splitter-graph-5d4cc-9fb6994f-tzflz	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
12:29:40	kserve-ci-e2e-test	kubelet	success-200-isvc-beb62-predictor-74847944ff-snqr2	Killing	Stopping container kserve-container
12:29:40	kserve-ci-e2e-test	kubelet	error-404-isvc-beb62-predictor-64dc4bbbb-759qg	Killing	Stopping container kube-rbac-proxy
12:29:40	kserve-ci-e2e-test	kubelet	error-404-isvc-beb62-predictor-64dc4bbbb-759qg	Killing	Stopping container kserve-container
12:29:40	kserve-ci-e2e-test	kubelet	switch-graph-beb62-5c7d9675f8-nbnkw	Killing	Stopping container switch-graph-beb62
12:29:40	kserve-ci-e2e-test	kubelet	success-200-isvc-beb62-predictor-74847944ff-snqr2	Killing	Stopping container kube-rbac-proxy
12:30:05 (x5)	kserve-ci-e2e-test	kubelet	switch-graph-beb62-5c7d9675f8-nbnkw	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503