Events

Time	Namespace	Component	RelatedObject	Reason	Message
21:23:06	kserve-ci-e2e-test		splitter-graph-a1ff2-574bdd87b5-hktnd	Scheduled	Successfully assigned kserve-ci-e2e-test/splitter-graph-a1ff2-574bdd87b5-hktnd to ip-10-0-139-55.ec2.internal
21:00:23	kserve-ci-e2e-test		success-200-isvc-23363-predictor-5875556545-7gqbn	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-23363-predictor-5875556545-7gqbn to ip-10-0-139-55.ec2.internal
21:01:20	kserve-ci-e2e-test		error-404-isvc-ed647-predictor-7fbb665cd-xwzbp	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-ed647-predictor-7fbb665cd-xwzbp to ip-10-0-138-134.ec2.internal
20:58:48	kserve-ci-e2e-test		success-200-isvc-a625d-predictor-68cf5d77c9-k7djz	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-a625d-predictor-68cf5d77c9-k7djz to ip-10-0-139-55.ec2.internal
21:21:56	kserve-ci-e2e-test		success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v to ip-10-0-139-55.ec2.internal
21:10:45	kserve-ci-e2e-test		error-404-isvc-62e6f-predictor-6984c9779b-6qqq9	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-62e6f-predictor-6984c9779b-6qqq9 to ip-10-0-138-134.ec2.internal
20:58:48	kserve-ci-e2e-test		isvc-xgboost-graph-predictor-555fc69d56-f4cj5	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-f4cj5 to ip-10-0-140-118.ec2.internal
21:12:07	kserve-ci-e2e-test		error-404-isvc-b5a48-predictor-5bb6b4f9f8-4b79b	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-b5a48-predictor-5bb6b4f9f8-4b79b to ip-10-0-138-134.ec2.internal
21:21:45	kserve-ci-e2e-test		splitter-graph-d3600-78645b4dcf-b5xv7	Scheduled	Successfully assigned kserve-ci-e2e-test/splitter-graph-d3600-78645b4dcf-b5xv7 to ip-10-0-139-55.ec2.internal
21:10:45	kserve-ci-e2e-test		success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r to ip-10-0-139-55.ec2.internal
21:09:48	kserve-ci-e2e-test		error-404-isvc-46851-predictor-7f76877c8-jkqxj	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-46851-predictor-7f76877c8-jkqxj to ip-10-0-138-134.ec2.internal
21:01:08	kserve-ci-e2e-test		model-chainer-58db8ff9c8-zlxhc	Scheduled	Successfully assigned kserve-ci-e2e-test/model-chainer-58db8ff9c8-zlxhc to ip-10-0-139-55.ec2.internal
21:11:09	kserve-ci-e2e-test		error-404-isvc-31e6a-predictor-764bc8c74b-kd5vf	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-31e6a-predictor-764bc8c74b-kd5vf to ip-10-0-138-134.ec2.internal
21:09:48	kserve-ci-e2e-test		success-200-isvc-46851-predictor-5c8c955cdc-c8zks	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-46851-predictor-5c8c955cdc-c8zks to ip-10-0-139-55.ec2.internal
21:12:19	kserve-ci-e2e-test		ensemble-graph-31e6a-b7ddd645d-m4f2b	Scheduled	Successfully assigned kserve-ci-e2e-test/ensemble-graph-31e6a-b7ddd645d-m4f2b to ip-10-0-139-55.ec2.internal
21:11:55	kserve-ci-e2e-test		sequence-graph-62e6f-7c8567c89d-p74m9	Scheduled	Successfully assigned kserve-ci-e2e-test/sequence-graph-62e6f-7c8567c89d-p74m9 to ip-10-0-139-55.ec2.internal
21:00:08	kserve-ci-e2e-test		switch-graph-a625d-77f5ffd85c-nwwfg	Scheduled	Successfully assigned kserve-ci-e2e-test/switch-graph-a625d-77f5ffd85c-nwwfg to ip-10-0-139-55.ec2.internal
21:21:32	kserve-ci-e2e-test		error-404-isvc-3ccd0-predictor-5c789f8bd8-c7nfl	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-3ccd0-predictor-5c789f8bd8-c7nfl to ip-10-0-138-134.ec2.internal
21:01:33	kserve-ci-e2e-test		switch-graph-23363-67f7d4f7ff-mp2w6	Scheduled	Successfully assigned kserve-ci-e2e-test/switch-graph-23363-67f7d4f7ff-mp2w6 to ip-10-0-139-55.ec2.internal
21:00:23	kserve-ci-e2e-test		error-404-isvc-23363-predictor-bbcb4956-hk5qt	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-23363-predictor-bbcb4956-hk5qt to ip-10-0-138-134.ec2.internal
21:13:17	kserve-ci-e2e-test		sequence-graph-b5a48-848c5fd7d8-nzfp4	Scheduled	Successfully assigned kserve-ci-e2e-test/sequence-graph-b5a48-848c5fd7d8-nzfp4 to ip-10-0-139-55.ec2.internal
21:02:29	kserve-ci-e2e-test		sequence-graph-ed647-5445897977-p9svl	Scheduled	Successfully assigned kserve-ci-e2e-test/sequence-graph-ed647-5445897977-p9svl to ip-10-0-139-55.ec2.internal
20:58:48	kserve-ci-e2e-test		isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps to ip-10-0-140-118.ec2.internal
21:10:58	kserve-ci-e2e-test		ensemble-graph-46851-555c4ddb89-n7mwm	Scheduled	Successfully assigned kserve-ci-e2e-test/ensemble-graph-46851-555c4ddb89-n7mwm to ip-10-0-139-55.ec2.internal
21:22:42	kserve-ci-e2e-test		switch-graph-3ccd0-6464d894c-mplj9	Scheduled	Successfully assigned kserve-ci-e2e-test/switch-graph-3ccd0-6464d894c-mplj9 to ip-10-0-139-55.ec2.internal
21:21:56	kserve-ci-e2e-test		error-404-isvc-a1ff2-predictor-6bd48997f4-pp8xj	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-a1ff2-predictor-6bd48997f4-pp8xj to ip-10-0-138-134.ec2.internal
21:21:32	kserve-ci-e2e-test		success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg to ip-10-0-139-55.ec2.internal
20:58:49	kserve-ci-e2e-test		isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d	Scheduled	Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d to ip-10-0-140-118.ec2.internal
21:01:20	kserve-ci-e2e-test		success-200-isvc-ed647-predictor-7fdb57868c-fgq77	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-ed647-predictor-7fdb57868c-fgq77 to ip-10-0-139-55.ec2.internal
21:20:35	kserve-ci-e2e-test		success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw to ip-10-0-139-55.ec2.internal
21:12:07	kserve-ci-e2e-test		success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd to ip-10-0-139-55.ec2.internal
21:20:35	kserve-ci-e2e-test		error-404-isvc-d3600-predictor-5bc9bb5464-rns9p	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-d3600-predictor-5bc9bb5464-rns9p to ip-10-0-138-134.ec2.internal
21:11:09	kserve-ci-e2e-test		success-200-isvc-31e6a-predictor-6c64485579-ztxgg	Scheduled	Successfully assigned kserve-ci-e2e-test/success-200-isvc-31e6a-predictor-6c64485579-ztxgg to ip-10-0-139-55.ec2.internal
20:58:48	kserve-ci-e2e-test		error-404-isvc-a625d-predictor-7679674998-2mt57	Scheduled	Successfully assigned kserve-ci-e2e-test/error-404-isvc-a625d-predictor-7679674998-2mt57 to ip-10-0-138-134.ec2.internal
20:58:48	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-a625d-predictor-68cf5d77c9	SuccessfulCreate	Created pod: success-200-isvc-a625d-predictor-68cf5d77c9-k7djz
20:58:48	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-graph-1-predictor-64d8fbb9c6	SuccessfulCreate	Created pod: isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps
20:58:48	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-graph-1-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-graph-1-predictor-64d8fbb9c6 from 0 to 1
20:58:48	kserve-ci-e2e-test	kubelet	error-404-isvc-a625d-predictor-7679674998-2mt57	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "error-404-isvc-a625d-predictor-serving-cert" not found
20:58:48	kserve-ci-e2e-test	deployment-controller	isvc-sklearn-graph-2-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-graph-2-predictor-74b67d48cc from 0 to 1
20:58:48	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-a625d-predictor-7679674998	SuccessfulCreate	Created pod: error-404-isvc-a625d-predictor-7679674998-2mt57
20:58:48	kserve-ci-e2e-test	deployment-controller	error-404-isvc-a625d-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-a625d-predictor-7679674998 from 0 to 1
20:58:48	kserve-ci-e2e-test	deployment-controller	isvc-xgboost-graph-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-graph-predictor-555fc69d56 from 0 to 1
20:58:48	kserve-ci-e2e-test	deployment-controller	success-200-isvc-a625d-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-a625d-predictor-68cf5d77c9 from 0 to 1
20:58:48	kserve-ci-e2e-test	replicaset-controller	isvc-xgboost-graph-predictor-555fc69d56	SuccessfulCreate	Created pod: isvc-xgboost-graph-predictor-555fc69d56-f4cj5
20:58:48	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
20:58:49	kserve-ci-e2e-test	kubelet	error-404-isvc-a625d-predictor-7679674998-2mt57	Pulling	Pulling image "quay.io/opendatahub/error-404-isvc:odh-pr-1999"
20:58:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d	Started	Started container storage-initializer
20:58:49	kserve-ci-e2e-test	replicaset-controller	isvc-sklearn-graph-2-predictor-74b67d48cc	SuccessfulCreate	Created pod: isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d
20:58:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d	Created	Created container: storage-initializer
20:58:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:088810e21187103d825698143fc93d3ba94ca1f9b7facd7cf4b1d40af2e9f6fb" already present on machine
20:58:49	kserve-ci-e2e-test	multus	isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d	AddedInterface	Add eth0 [10.132.0.29/23] from ovn-kubernetes
20:58:49	kserve-ci-e2e-test	multus	error-404-isvc-a625d-predictor-7679674998-2mt57	AddedInterface	Add eth0 [10.133.0.19/23] from ovn-kubernetes
20:58:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps	Created	Created container: storage-initializer
20:58:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps	Started	Started container storage-initializer
20:58:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4cj5	Started	Started container storage-initializer
20:58:49	kserve-ci-e2e-test	kubelet	success-200-isvc-a625d-predictor-68cf5d77c9-k7djz	Pulling	Pulling image "quay.io/opendatahub/success-200-isvc:odh-pr-1999"
20:58:49	kserve-ci-e2e-test	multus	success-200-isvc-a625d-predictor-68cf5d77c9-k7djz	AddedInterface	Add eth0 [10.134.0.30/23] from ovn-kubernetes
20:58:49	kserve-ci-e2e-test	multus	isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps	AddedInterface	Add eth0 [10.132.0.27/23] from ovn-kubernetes
20:58:49	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:088810e21187103d825698143fc93d3ba94ca1f9b7facd7cf4b1d40af2e9f6fb" already present on machine
20:58:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4cj5	Created	Created container: storage-initializer
20:58:49	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4cj5	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:088810e21187103d825698143fc93d3ba94ca1f9b7facd7cf4b1d40af2e9f6fb" already present on machine
20:58:49	kserve-ci-e2e-test	multus	isvc-xgboost-graph-predictor-555fc69d56-f4cj5	AddedInterface	Add eth0 [10.132.0.28/23] from ovn-kubernetes
20:58:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1999"
20:58:53	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1999"
20:58:53	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4cj5	Pulling	Pulling image "kserve/xgbserver:latest"
20:59:00	kserve-ci-e2e-test	kubelet	success-200-isvc-a625d-predictor-68cf5d77c9-k7djz	Pulled	Successfully pulled image "quay.io/opendatahub/success-200-isvc:odh-pr-1999" in 11.856s (11.856s including waiting). Image size: 1385941582 bytes.
20:59:01	kserve-ci-e2e-test	kubelet	success-200-isvc-a625d-predictor-68cf5d77c9-k7djz	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
20:59:01	kserve-ci-e2e-test	kubelet	success-200-isvc-a625d-predictor-68cf5d77c9-k7djz	Created	Created container: kserve-container
20:59:01	kserve-ci-e2e-test	kubelet	success-200-isvc-a625d-predictor-68cf5d77c9-k7djz	Started	Started container kserve-container
20:59:02	kserve-ci-e2e-test	kubelet	error-404-isvc-a625d-predictor-7679674998-2mt57	Pulled	Successfully pulled image "quay.io/opendatahub/error-404-isvc:odh-pr-1999" in 12.234s (12.234s including waiting). Image size: 1386042952 bytes.
20:59:02	kserve-ci-e2e-test	kubelet	error-404-isvc-a625d-predictor-7679674998-2mt57	Created	Created container: kserve-container
20:59:02	kserve-ci-e2e-test	kubelet	error-404-isvc-a625d-predictor-7679674998-2mt57	Started	Started container kserve-container
20:59:02	kserve-ci-e2e-test	kubelet	error-404-isvc-a625d-predictor-7679674998-2mt57	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
20:59:03	kserve-ci-e2e-test	kubelet	success-200-isvc-a625d-predictor-68cf5d77c9-k7djz	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.311s (2.311s including waiting). Image size: 211946088 bytes.
20:59:03	kserve-ci-e2e-test	kubelet	success-200-isvc-a625d-predictor-68cf5d77c9-k7djz	Created	Created container: kube-rbac-proxy
20:59:03	kserve-ci-e2e-test	kubelet	success-200-isvc-a625d-predictor-68cf5d77c9-k7djz	Started	Started container kube-rbac-proxy
20:59:04	kserve-ci-e2e-test	kubelet	error-404-isvc-a625d-predictor-7679674998-2mt57	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.139s (2.139s including waiting). Image size: 211946088 bytes.
20:59:04	kserve-ci-e2e-test	kubelet	error-404-isvc-a625d-predictor-7679674998-2mt57	Created	Created container: kube-rbac-proxy
20:59:04	kserve-ci-e2e-test	kubelet	error-404-isvc-a625d-predictor-7679674998-2mt57	Started	Started container kube-rbac-proxy
20:59:18 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-a625d-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:59:18 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-a625d-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:59:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1999" in 29.536s (29.536s including waiting). Image size: 1604543374 bytes.
20:59:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4cj5	Started	Started container kserve-container
20:59:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps	Started	Started container kserve-container
20:59:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps	Created	Created container: kserve-container
20:59:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
20:59:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1999" in 29.539s (29.539s including waiting). Image size: 1604543374 bytes.
20:59:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d	Created	Created container: kserve-container
20:59:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d	Started	Started container kserve-container
20:59:23	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
20:59:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4cj5	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 30.059s (30.059s including waiting). Image size: 1435742980 bytes.
20:59:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4cj5	Created	Created container: kserve-container
20:59:23	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4cj5	Pulling	Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3"
20:59:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.84s (2.84s including waiting). Image size: 211946088 bytes.
20:59:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps	Created	Created container: kube-rbac-proxy
20:59:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.894s (2.894s including waiting). Image size: 211946088 bytes.
20:59:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d	Created	Created container: kube-rbac-proxy
20:59:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4cj5	Created	Created container: kube-rbac-proxy
20:59:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4cj5	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.45s (2.45s including waiting). Image size: 211946088 bytes.
20:59:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps	Started	Started container kube-rbac-proxy
20:59:26	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d	Started	Started container kube-rbac-proxy
20:59:26	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4cj5	Started	Started container kube-rbac-proxy
20:59:39 (x6)	kserve-ci-e2e-test	kubelet	success-200-isvc-a625d-predictor-68cf5d77c9-k7djz	Unhealthy	Readiness probe failed: dial tcp 10.134.0.30:8080: connect: connection refused
20:59:48 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-a625d-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:59:48 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-a625d-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
20:59:49	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-a625d	InferenceServiceReady	InferenceService [success-200-isvc-a625d] is Ready
20:59:49 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-a625d	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
20:59:50 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-a625d-predictor-7679674998-2mt57	Unhealthy	Readiness probe failed: dial tcp 10.133.0.19:8080: connect: connection refused
21:00:00 (x5)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-a625d	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:00:00	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-a625d	InferenceServiceReady	InferenceService [error-404-isvc-a625d] is Ready
21:00:03 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:00:03 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:00:03 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:00:03 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:00:03 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:00:03 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:00:08	kserve-ci-e2e-test	InferenceGraphController	switch-graph-a625d	InferenceGraphReady	InferenceGraph [switch-graph-a625d] is Ready
21:00:08	kserve-ci-e2e-test	deployment-controller	switch-graph-a625d	ScalingReplicaSet	Scaled up replica set switch-graph-a625d-77f5ffd85c from 0 to 1
21:00:08	kserve-ci-e2e-test	replicaset-controller	switch-graph-a625d-77f5ffd85c	SuccessfulCreate	Created pod: switch-graph-a625d-77f5ffd85c-nwwfg
21:00:08	kserve-ci-e2e-test	kubelet	switch-graph-a625d-77f5ffd85c-nwwfg	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "switch-graph-a625d-serving-cert" not found
21:00:09	kserve-ci-e2e-test	kubelet	switch-graph-a625d-77f5ffd85c-nwwfg	Pulling	Pulling image "quay.io/opendatahub/kserve-router@sha256:2c5833579293a5043b5db03d2dc3bee6dec8d8c7a284a9a790353f02c7a72dd3"
21:00:09	kserve-ci-e2e-test	multus	switch-graph-a625d-77f5ffd85c-nwwfg	AddedInterface	Add eth0 [10.134.0.31/23] from ovn-kubernetes
21:00:11	kserve-ci-e2e-test	kubelet	switch-graph-a625d-77f5ffd85c-nwwfg	Created	Created container: switch-graph-a625d
21:00:11	kserve-ci-e2e-test	kubelet	switch-graph-a625d-77f5ffd85c-nwwfg	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:2c5833579293a5043b5db03d2dc3bee6dec8d8c7a284a9a790353f02c7a72dd3" in 2.025s (2.025s including waiting). Image size: 211751499 bytes.
21:00:11	kserve-ci-e2e-test	kubelet	switch-graph-a625d-77f5ffd85c-nwwfg	Started	Started container switch-graph-a625d
21:00:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-a625d-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:00:18 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-a625d-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:00:18 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-a625d-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:00:18 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-a625d-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:00:23	kserve-ci-e2e-test	kubelet	error-404-isvc-a625d-predictor-7679674998-2mt57	Killing	Stopping container kserve-container
21:00:23	kserve-ci-e2e-test	deployment-controller	error-404-isvc-23363-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-23363-predictor-bbcb4956 from 0 to 1
21:00:23	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-a625d	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:00:23	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-a625d	FailedUpdateStatus	horizontalpodautoscalers.autoscaling "switch-graph-a625d" not found
21:00:23	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-a625d	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:00:23	kserve-ci-e2e-test	kubelet	switch-graph-a625d-77f5ffd85c-nwwfg	Killing	Stopping container switch-graph-a625d
21:00:23	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-23363-predictor-5875556545	SuccessfulCreate	Created pod: success-200-isvc-23363-predictor-5875556545-7gqbn
21:00:23	kserve-ci-e2e-test	kubelet	error-404-isvc-a625d-predictor-7679674998-2mt57	Killing	Stopping container kube-rbac-proxy
21:00:23	kserve-ci-e2e-test	deployment-controller	success-200-isvc-23363-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-23363-predictor-5875556545 from 0 to 1
21:00:23	kserve-ci-e2e-test	kubelet	error-404-isvc-23363-predictor-bbcb4956-hk5qt	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "error-404-isvc-23363-predictor-serving-cert" not found
21:00:23	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-23363-predictor-bbcb4956	SuccessfulCreate	Created pod: error-404-isvc-23363-predictor-bbcb4956-hk5qt
21:00:23	kserve-ci-e2e-test	kubelet	success-200-isvc-a625d-predictor-68cf5d77c9-k7djz	Killing	Stopping container kserve-container
21:00:23	kserve-ci-e2e-test	kubelet	success-200-isvc-a625d-predictor-68cf5d77c9-k7djz	Killing	Stopping container kube-rbac-proxy
21:00:24	kserve-ci-e2e-test	kubelet	error-404-isvc-23363-predictor-bbcb4956-hk5qt	Started	Started container kserve-container
21:00:24	kserve-ci-e2e-test	kubelet	success-200-isvc-23363-predictor-5875556545-7gqbn	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1999" already present on machine
21:00:24	kserve-ci-e2e-test	kubelet	success-200-isvc-23363-predictor-5875556545-7gqbn	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:00:24	kserve-ci-e2e-test	kubelet	success-200-isvc-23363-predictor-5875556545-7gqbn	Created	Created container: kube-rbac-proxy
21:00:24	kserve-ci-e2e-test	kubelet	success-200-isvc-23363-predictor-5875556545-7gqbn	Started	Started container kube-rbac-proxy
21:00:24	kserve-ci-e2e-test	kubelet	success-200-isvc-a625d-predictor-68cf5d77c9-k7djz	Unhealthy	Readiness probe failed: Get "https://10.134.0.30:8643/healthz": dial tcp 10.134.0.30:8643: connect: connection refused
21:00:24	kserve-ci-e2e-test	multus	success-200-isvc-23363-predictor-5875556545-7gqbn	AddedInterface	Add eth0 [10.134.0.32/23] from ovn-kubernetes
21:00:24	kserve-ci-e2e-test	kubelet	error-404-isvc-23363-predictor-bbcb4956-hk5qt	Started	Started container kube-rbac-proxy
21:00:24	kserve-ci-e2e-test	kubelet	error-404-isvc-23363-predictor-bbcb4956-hk5qt	Created	Created container: kube-rbac-proxy
21:00:24	kserve-ci-e2e-test	kubelet	error-404-isvc-23363-predictor-bbcb4956-hk5qt	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:00:24	kserve-ci-e2e-test	kubelet	success-200-isvc-23363-predictor-5875556545-7gqbn	Created	Created container: kserve-container
21:00:24	kserve-ci-e2e-test	kubelet	error-404-isvc-23363-predictor-bbcb4956-hk5qt	Created	Created container: kserve-container
21:00:24	kserve-ci-e2e-test	kubelet	error-404-isvc-23363-predictor-bbcb4956-hk5qt	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1999" already present on machine
21:00:24	kserve-ci-e2e-test	multus	error-404-isvc-23363-predictor-bbcb4956-hk5qt	AddedInterface	Add eth0 [10.133.0.20/23] from ovn-kubernetes
21:00:24	kserve-ci-e2e-test	kubelet	success-200-isvc-23363-predictor-5875556545-7gqbn	Started	Started container kserve-container
21:00:25	kserve-ci-e2e-test	kubelet	error-404-isvc-a625d-predictor-7679674998-2mt57	Unhealthy	Readiness probe failed: Get "https://10.133.0.19:8643/healthz": dial tcp 10.133.0.19:8643: connect: connection refused
21:00:32	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-graph	InferenceServiceReady	InferenceService [isvc-xgboost-graph] is Ready
21:00:33 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-xgboost-graph	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:00:36 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps	Unhealthy	Readiness probe failed: dial tcp 10.132.0.27:8080: connect: connection refused
21:00:42	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-2	InferenceServiceReady	InferenceService [isvc-sklearn-graph-2] is Ready
21:00:43 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:00:46 (x9)	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:00:46	kserve-ci-e2e-test	v1beta1Controllers	isvc-sklearn-graph-1	InferenceServiceReady	InferenceService [isvc-sklearn-graph-1] is Ready
21:00:53 (x6)	kserve-ci-e2e-test	kubelet	switch-graph-a625d-77f5ffd85c-nwwfg	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
21:00:53 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-23363-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:00:53 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-23363-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:00:53 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-23363-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:00:53 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-23363-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:01:03 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:01:03 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:01:03 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:01:03 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:01:04 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:01:04 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:01:08	kserve-ci-e2e-test	deployment-controller	model-chainer	ScalingReplicaSet	Scaled up replica set model-chainer-58db8ff9c8 from 0 to 1
21:01:08	kserve-ci-e2e-test	InferenceGraphController	model-chainer	InferenceGraphReady	InferenceGraph [model-chainer] is Ready
21:01:08	kserve-ci-e2e-test	multus	model-chainer-58db8ff9c8-zlxhc	AddedInterface	Add eth0 [10.134.0.33/23] from ovn-kubernetes
21:01:08	kserve-ci-e2e-test	kubelet	model-chainer-58db8ff9c8-zlxhc	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:2c5833579293a5043b5db03d2dc3bee6dec8d8c7a284a9a790353f02c7a72dd3" already present on machine
21:01:08	kserve-ci-e2e-test	kubelet	model-chainer-58db8ff9c8-zlxhc	Created	Created container: model-chainer
21:01:08	kserve-ci-e2e-test	kubelet	model-chainer-58db8ff9c8-zlxhc	Started	Started container model-chainer
21:01:08	kserve-ci-e2e-test	replicaset-controller	model-chainer-58db8ff9c8	SuccessfulCreate	Created pod: model-chainer-58db8ff9c8-zlxhc
21:01:19	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-ed647-predictor-7fdb57868c	SuccessfulCreate	Created pod: success-200-isvc-ed647-predictor-7fdb57868c-fgq77
21:01:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps	Killing	Stopping container kube-rbac-proxy
21:01:19	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4cj5	Killing	Stopping container kserve-container
21:01:19	kserve-ci-e2e-test	deployment-controller	success-200-isvc-ed647-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-ed647-predictor-7fdb57868c from 0 to 1
21:01:19	kserve-ci-e2e-test	kubelet	model-chainer-58db8ff9c8-zlxhc	Killing	Stopping container model-chainer
21:01:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d	Killing	Stopping container kube-rbac-proxy
21:01:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d	Killing	Stopping container kserve-container
21:01:19	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps	Killing	Stopping container kserve-container
21:01:19	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4cj5	Killing	Stopping container kube-rbac-proxy
21:01:20	kserve-ci-e2e-test	kubelet	error-404-isvc-ed647-predictor-7fbb665cd-xwzbp	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "error-404-isvc-ed647-predictor-serving-cert" not found
21:01:20	kserve-ci-e2e-test	deployment-controller	error-404-isvc-ed647-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-ed647-predictor-7fbb665cd from 0 to 1
21:01:20 (x6)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-23363	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:01:20	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-23363	InferenceServiceReady	InferenceService [success-200-isvc-23363] is Ready
21:01:20	kserve-ci-e2e-test	kubelet	success-200-isvc-ed647-predictor-7fdb57868c-fgq77	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "success-200-isvc-ed647-predictor-serving-cert" not found
21:01:20	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-ed647-predictor-7fbb665cd	SuccessfulCreate	Created pod: error-404-isvc-ed647-predictor-7fbb665cd-xwzbp
21:01:21	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-23363	InferenceServiceReady	InferenceService [error-404-isvc-23363] is Ready
21:01:21	kserve-ci-e2e-test	multus	success-200-isvc-ed647-predictor-7fdb57868c-fgq77	AddedInterface	Add eth0 [10.134.0.34/23] from ovn-kubernetes
21:01:21	kserve-ci-e2e-test	kubelet	success-200-isvc-ed647-predictor-7fdb57868c-fgq77	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:01:21	kserve-ci-e2e-test	kubelet	success-200-isvc-ed647-predictor-7fdb57868c-fgq77	Created	Created container: kube-rbac-proxy
21:01:21	kserve-ci-e2e-test	kubelet	success-200-isvc-ed647-predictor-7fdb57868c-fgq77	Started	Started container kube-rbac-proxy
21:01:21	kserve-ci-e2e-test	kubelet	success-200-isvc-ed647-predictor-7fdb57868c-fgq77	Created	Created container: kserve-container
21:01:21	kserve-ci-e2e-test	kubelet	success-200-isvc-ed647-predictor-7fdb57868c-fgq77	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1999" already present on machine
21:01:21	kserve-ci-e2e-test	kubelet	success-200-isvc-ed647-predictor-7fdb57868c-fgq77	Started	Started container kserve-container
21:01:21 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-23363	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:01:21	kserve-ci-e2e-test	kubelet	error-404-isvc-ed647-predictor-7fbb665cd-xwzbp	Started	Started container kube-rbac-proxy
21:01:21	kserve-ci-e2e-test	kubelet	error-404-isvc-ed647-predictor-7fbb665cd-xwzbp	Created	Created container: kube-rbac-proxy
21:01:21	kserve-ci-e2e-test	kubelet	error-404-isvc-ed647-predictor-7fbb665cd-xwzbp	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:01:21	kserve-ci-e2e-test	multus	error-404-isvc-ed647-predictor-7fbb665cd-xwzbp	AddedInterface	Add eth0 [10.133.0.21/23] from ovn-kubernetes
21:01:21	kserve-ci-e2e-test	kubelet	error-404-isvc-ed647-predictor-7fbb665cd-xwzbp	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1999" already present on machine
21:01:21	kserve-ci-e2e-test	kubelet	error-404-isvc-ed647-predictor-7fbb665cd-xwzbp	Created	Created container: kserve-container
21:01:21	kserve-ci-e2e-test	kubelet	error-404-isvc-ed647-predictor-7fbb665cd-xwzbp	Started	Started container kserve-container
21:01:22	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4cj5	Unhealthy	Readiness probe failed: Get "https://10.132.0.28:8643/healthz": dial tcp 10.132.0.28:8643: connect: connection refused
21:01:22 (x9)	kserve-ci-e2e-test	kubelet	isvc-xgboost-graph-predictor-555fc69d56-f4cj5	Unhealthy	Readiness probe failed: dial tcp 10.132.0.28:8080: connect: connection refused
21:01:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-1-predictor-64d8fbb9c6-m4mps	Unhealthy	Readiness probe failed: Get "https://10.132.0.27:8643/healthz": dial tcp 10.132.0.27:8643: connect: connection refused
21:01:22 (x10)	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d	Unhealthy	Readiness probe failed: dial tcp 10.132.0.29:8080: connect: connection refused
21:01:22	kserve-ci-e2e-test	kubelet	isvc-sklearn-graph-2-predictor-74b67d48cc-hsw8d	Unhealthy	Readiness probe failed: Get "https://10.132.0.29:8643/healthz": dial tcp 10.132.0.29:8643: connect: connection refused
21:01:33	kserve-ci-e2e-test	InferenceGraphController	switch-graph-23363	InferenceGraphReady	InferenceGraph [switch-graph-23363] is Ready
21:01:33	kserve-ci-e2e-test	deployment-controller	switch-graph-23363	ScalingReplicaSet	Scaled up replica set switch-graph-23363-67f7d4f7ff from 0 to 1
21:01:33	kserve-ci-e2e-test	kubelet	switch-graph-23363-67f7d4f7ff-mp2w6	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:2c5833579293a5043b5db03d2dc3bee6dec8d8c7a284a9a790353f02c7a72dd3" already present on machine
21:01:33	kserve-ci-e2e-test	multus	switch-graph-23363-67f7d4f7ff-mp2w6	AddedInterface	Add eth0 [10.134.0.35/23] from ovn-kubernetes
21:01:33	kserve-ci-e2e-test	replicaset-controller	switch-graph-23363-67f7d4f7ff	SuccessfulCreate	Created pod: switch-graph-23363-67f7d4f7ff-mp2w6
21:01:34	kserve-ci-e2e-test	kubelet	switch-graph-23363-67f7d4f7ff-mp2w6	Started	Started container switch-graph-23363
21:01:34	kserve-ci-e2e-test	kubelet	switch-graph-23363-67f7d4f7ff-mp2w6	Created	Created container: switch-graph-23363
21:01:35	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-ed647-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:01:35	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-ed647-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:01:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-23363-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:01:38 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-23363-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:01:45 (x6)	kserve-ci-e2e-test	kubelet	model-chainer-58db8ff9c8-zlxhc	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
21:01:48	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-23363	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:01:48	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-23363	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:01:53 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-23363-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:01:53 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-23363-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:02:05 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-ed647-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:02:05 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-ed647-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:02:09 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-ed647-predictor-7fbb665cd-xwzbp	Unhealthy	Readiness probe failed: dial tcp 10.133.0.21:8080: connect: connection refused
21:02:18	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-ed647	InferenceServiceReady	InferenceService [success-200-isvc-ed647] is Ready
21:02:18 (x6)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-ed647	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:02:18 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-23363	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:02:18 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-23363	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:02:19 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-ed647	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:02:19	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-ed647	InferenceServiceReady	InferenceService [error-404-isvc-ed647] is Ready
21:02:29	kserve-ci-e2e-test	replicaset-controller	sequence-graph-ed647-5445897977	SuccessfulCreate	Created pod: sequence-graph-ed647-5445897977-p9svl
21:02:29	kserve-ci-e2e-test	deployment-controller	sequence-graph-ed647	ScalingReplicaSet	Scaled up replica set sequence-graph-ed647-5445897977 from 0 to 1
21:02:30	kserve-ci-e2e-test	multus	sequence-graph-ed647-5445897977-p9svl	AddedInterface	Add eth0 [10.134.0.36/23] from ovn-kubernetes
21:02:30	kserve-ci-e2e-test	kubelet	sequence-graph-ed647-5445897977-p9svl	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:2c5833579293a5043b5db03d2dc3bee6dec8d8c7a284a9a790353f02c7a72dd3" already present on machine
21:02:30	kserve-ci-e2e-test	kubelet	sequence-graph-ed647-5445897977-p9svl	Started	Started container sequence-graph-ed647
21:02:30	kserve-ci-e2e-test	kubelet	sequence-graph-ed647-5445897977-p9svl	Created	Created container: sequence-graph-ed647
21:02:30	kserve-ci-e2e-test	InferenceGraphController	sequence-graph-ed647	InferenceGraphReady	InferenceGraph [sequence-graph-ed647] is Ready
21:02:50 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-ed647-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:02:50 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-ed647-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:02:50 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-ed647-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:02:50 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-ed647-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:03:15 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-ed647	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:03:15 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-ed647	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:09:48	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-46851-predictor-7f76877c8	SuccessfulCreate	Created pod: error-404-isvc-46851-predictor-7f76877c8-jkqxj
21:09:48	kserve-ci-e2e-test	kubelet	error-404-isvc-46851-predictor-7f76877c8-jkqxj	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "error-404-isvc-46851-predictor-serving-cert" not found
21:09:48	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-46851-predictor-5c8c955cdc	SuccessfulCreate	Created pod: success-200-isvc-46851-predictor-5c8c955cdc-c8zks
21:09:48	kserve-ci-e2e-test	deployment-controller	success-200-isvc-46851-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-46851-predictor-5c8c955cdc from 0 to 1
21:09:48	kserve-ci-e2e-test	kubelet	success-200-isvc-23363-predictor-5875556545-7gqbn	Killing	Stopping container kube-rbac-proxy
21:09:48	kserve-ci-e2e-test	deployment-controller	error-404-isvc-46851-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-46851-predictor-7f76877c8 from 0 to 1
21:09:48	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-46851	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-46851": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-46851": the object has been modified; please apply your changes to the latest version and try again
21:09:48	kserve-ci-e2e-test	kubelet	error-404-isvc-23363-predictor-bbcb4956-hk5qt	Killing	Stopping container kserve-container
21:09:48	kserve-ci-e2e-test	kubelet	success-200-isvc-23363-predictor-5875556545-7gqbn	Killing	Stopping container kserve-container
21:09:48	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-46851	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-46851": the object has been modified; please apply your changes to the latest version and try again
21:09:48	kserve-ci-e2e-test	kubelet	switch-graph-23363-67f7d4f7ff-mp2w6	Killing	Stopping container switch-graph-23363
21:09:48	kserve-ci-e2e-test	kubelet	error-404-isvc-23363-predictor-bbcb4956-hk5qt	Killing	Stopping container kube-rbac-proxy
21:09:49	kserve-ci-e2e-test	kubelet	success-200-isvc-46851-predictor-5c8c955cdc-c8zks	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1999" already present on machine
21:09:49	kserve-ci-e2e-test	multus	error-404-isvc-46851-predictor-7f76877c8-jkqxj	AddedInterface	Add eth0 [10.133.0.22/23] from ovn-kubernetes
21:09:49	kserve-ci-e2e-test	multus	success-200-isvc-46851-predictor-5c8c955cdc-c8zks	AddedInterface	Add eth0 [10.134.0.37/23] from ovn-kubernetes
21:09:49	kserve-ci-e2e-test	kubelet	success-200-isvc-46851-predictor-5c8c955cdc-c8zks	Created	Created container: kserve-container
21:09:49	kserve-ci-e2e-test	kubelet	error-404-isvc-46851-predictor-7f76877c8-jkqxj	Created	Created container: kserve-container
21:09:49	kserve-ci-e2e-test	kubelet	success-200-isvc-46851-predictor-5c8c955cdc-c8zks	Started	Started container kube-rbac-proxy
21:09:49	kserve-ci-e2e-test	kubelet	success-200-isvc-46851-predictor-5c8c955cdc-c8zks	Created	Created container: kube-rbac-proxy
21:09:49	kserve-ci-e2e-test	kubelet	success-200-isvc-46851-predictor-5c8c955cdc-c8zks	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:09:49	kserve-ci-e2e-test	kubelet	success-200-isvc-46851-predictor-5c8c955cdc-c8zks	Started	Started container kserve-container
21:09:49	kserve-ci-e2e-test	kubelet	error-404-isvc-46851-predictor-7f76877c8-jkqxj	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1999" already present on machine
21:09:50	kserve-ci-e2e-test	kubelet	error-404-isvc-46851-predictor-7f76877c8-jkqxj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:09:50 (x8)	kserve-ci-e2e-test	kubelet	success-200-isvc-23363-predictor-5875556545-7gqbn	Unhealthy	Readiness probe failed: dial tcp 10.134.0.32:8080: connect: connection refused
21:09:50	kserve-ci-e2e-test	kubelet	error-404-isvc-46851-predictor-7f76877c8-jkqxj	Started	Started container kserve-container
21:09:50	kserve-ci-e2e-test	kubelet	error-404-isvc-46851-predictor-7f76877c8-jkqxj	Created	Created container: kube-rbac-proxy
21:09:50	kserve-ci-e2e-test	kubelet	error-404-isvc-46851-predictor-7f76877c8-jkqxj	Started	Started container kube-rbac-proxy
21:09:50	kserve-ci-e2e-test	kubelet	success-200-isvc-23363-predictor-5875556545-7gqbn	Unhealthy	Readiness probe failed: Get "https://10.134.0.32:8643/healthz": dial tcp 10.134.0.32:8643: connect: connection refused
21:09:51 (x8)	kserve-ci-e2e-test	kubelet	error-404-isvc-23363-predictor-bbcb4956-hk5qt	Unhealthy	Readiness probe failed: dial tcp 10.133.0.20:8080: connect: connection refused
21:09:51	kserve-ci-e2e-test	kubelet	error-404-isvc-23363-predictor-bbcb4956-hk5qt	Unhealthy	Readiness probe failed: Get "https://10.133.0.20:8643/healthz": dial tcp 10.133.0.20:8643: connect: connection refused
21:10:15 (x6)	kserve-ci-e2e-test	kubelet	switch-graph-23363-67f7d4f7ff-mp2w6	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
21:10:33 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-46851-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:10:33 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-46851-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:10:33 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-46851-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:10:33 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-46851-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:10:35 (x7)	kserve-ci-e2e-test	kubelet	success-200-isvc-46851-predictor-5c8c955cdc-c8zks	Unhealthy	Readiness probe failed: dial tcp 10.134.0.37:8080: connect: connection refused
21:10:36 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-46851-predictor-7f76877c8-jkqxj	Unhealthy	Readiness probe failed: dial tcp 10.133.0.22:8080: connect: connection refused
21:10:45	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-62e6f-predictor-6984c9779b	SuccessfulCreate	Created pod: error-404-isvc-62e6f-predictor-6984c9779b-6qqq9
21:10:45	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-46851	InferenceServiceReady	InferenceService [success-200-isvc-46851] is Ready
21:10:45	kserve-ci-e2e-test	kubelet	sequence-graph-ed647-5445897977-p9svl	Killing	Stopping container sequence-graph-ed647
21:10:45	kserve-ci-e2e-test	kubelet	success-200-isvc-ed647-predictor-7fdb57868c-fgq77	Killing	Stopping container kube-rbac-proxy
21:10:45	kserve-ci-e2e-test	kubelet	error-404-isvc-ed647-predictor-7fbb665cd-xwzbp	Killing	Stopping container kserve-container
21:10:45	kserve-ci-e2e-test	kubelet	error-404-isvc-ed647-predictor-7fbb665cd-xwzbp	Killing	Stopping container kube-rbac-proxy
21:10:45	kserve-ci-e2e-test	kubelet	success-200-isvc-ed647-predictor-7fdb57868c-fgq77	Killing	Stopping container kserve-container
21:10:45	kserve-ci-e2e-test	deployment-controller	success-200-isvc-62e6f-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-62e6f-predictor-545f99f9cc from 0 to 1
21:10:45	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-62e6f-predictor-545f99f9cc	SuccessfulCreate	Created pod: success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r
21:10:45	kserve-ci-e2e-test	deployment-controller	error-404-isvc-62e6f-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-62e6f-predictor-6984c9779b from 0 to 1
21:10:45	kserve-ci-e2e-test	kubelet	success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "success-200-isvc-62e6f-predictor-serving-cert" not found
21:10:46	kserve-ci-e2e-test	kubelet	error-404-isvc-62e6f-predictor-6984c9779b-6qqq9	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:10:46 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-46851	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:10:46	kserve-ci-e2e-test	kubelet	error-404-isvc-62e6f-predictor-6984c9779b-6qqq9	Started	Started container kserve-container
21:10:46	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-46851	InferenceServiceReady	InferenceService [error-404-isvc-46851] is Ready
21:10:46	kserve-ci-e2e-test	kubelet	error-404-isvc-62e6f-predictor-6984c9779b-6qqq9	Created	Created container: kube-rbac-proxy
21:10:46	kserve-ci-e2e-test	kubelet	error-404-isvc-62e6f-predictor-6984c9779b-6qqq9	Started	Started container kube-rbac-proxy
21:10:46	kserve-ci-e2e-test	kubelet	error-404-isvc-62e6f-predictor-6984c9779b-6qqq9	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1999" already present on machine
21:10:46 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-46851	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:10:46	kserve-ci-e2e-test	multus	success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r	AddedInterface	Add eth0 [10.134.0.38/23] from ovn-kubernetes
21:10:46	kserve-ci-e2e-test	kubelet	success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1999" already present on machine
21:10:46	kserve-ci-e2e-test	multus	error-404-isvc-62e6f-predictor-6984c9779b-6qqq9	AddedInterface	Add eth0 [10.133.0.23/23] from ovn-kubernetes
21:10:46	kserve-ci-e2e-test	kubelet	error-404-isvc-62e6f-predictor-6984c9779b-6qqq9	Created	Created container: kserve-container
21:10:47	kserve-ci-e2e-test	kubelet	success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r	Started	Started container kube-rbac-proxy
21:10:47	kserve-ci-e2e-test	kubelet	success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r	Created	Created container: kube-rbac-proxy
21:10:47	kserve-ci-e2e-test	kubelet	success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:10:47	kserve-ci-e2e-test	kubelet	success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r	Created	Created container: kserve-container
21:10:47	kserve-ci-e2e-test	kubelet	success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r	Started	Started container kserve-container
21:10:48	kserve-ci-e2e-test	kubelet	success-200-isvc-ed647-predictor-7fdb57868c-fgq77	Unhealthy	Readiness probe failed: Get "https://10.134.0.34:8643/healthz": dial tcp 10.134.0.34:8643: connect: connection refused
21:10:48 (x8)	kserve-ci-e2e-test	kubelet	success-200-isvc-ed647-predictor-7fdb57868c-fgq77	Unhealthy	Readiness probe failed: dial tcp 10.134.0.34:8080: connect: connection refused
21:10:58	kserve-ci-e2e-test	deployment-controller	ensemble-graph-46851	ScalingReplicaSet	Scaled up replica set ensemble-graph-46851-555c4ddb89 from 0 to 1
21:10:58	kserve-ci-e2e-test	replicaset-controller	ensemble-graph-46851-555c4ddb89	SuccessfulCreate	Created pod: ensemble-graph-46851-555c4ddb89-n7mwm
21:10:58	kserve-ci-e2e-test	InferenceGraphController	ensemble-graph-46851	InferenceGraphReady	InferenceGraph [ensemble-graph-46851] is Ready
21:10:59	kserve-ci-e2e-test	kubelet	ensemble-graph-46851-555c4ddb89-n7mwm	Started	Started container ensemble-graph-46851
21:10:59	kserve-ci-e2e-test	kubelet	ensemble-graph-46851-555c4ddb89-n7mwm	Created	Created container: ensemble-graph-46851
21:10:59	kserve-ci-e2e-test	kubelet	ensemble-graph-46851-555c4ddb89-n7mwm	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:2c5833579293a5043b5db03d2dc3bee6dec8d8c7a284a9a790353f02c7a72dd3" already present on machine
21:10:59	kserve-ci-e2e-test	multus	ensemble-graph-46851-555c4ddb89-n7mwm	AddedInterface	Add eth0 [10.134.0.39/23] from ovn-kubernetes
21:11:00	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-62e6f-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:11:00	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-62e6f-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:11:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-46851-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:11:03 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-46851-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:11:03 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-46851-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:11:03 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-46851-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:11:09	kserve-ci-e2e-test	deployment-controller	success-200-isvc-31e6a-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-31e6a-predictor-6c64485579 from 0 to 1
21:11:09	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-31e6a-predictor-764bc8c74b	SuccessfulCreate	Created pod: error-404-isvc-31e6a-predictor-764bc8c74b-kd5vf
21:11:09	kserve-ci-e2e-test	kubelet	success-200-isvc-46851-predictor-5c8c955cdc-c8zks	Killing	Stopping container kserve-container
21:11:09	kserve-ci-e2e-test	kubelet	success-200-isvc-46851-predictor-5c8c955cdc-c8zks	Killing	Stopping container kube-rbac-proxy
21:11:09	kserve-ci-e2e-test	kubelet	ensemble-graph-46851-555c4ddb89-n7mwm	Killing	Stopping container ensemble-graph-46851
21:11:09	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-31e6a-predictor-6c64485579	SuccessfulCreate	Created pod: success-200-isvc-31e6a-predictor-6c64485579-ztxgg
21:11:09	kserve-ci-e2e-test	kubelet	success-200-isvc-31e6a-predictor-6c64485579-ztxgg	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "success-200-isvc-31e6a-predictor-serving-cert" not found
21:11:09	kserve-ci-e2e-test	kubelet	error-404-isvc-46851-predictor-7f76877c8-jkqxj	Killing	Stopping container kube-rbac-proxy
21:11:09	kserve-ci-e2e-test	deployment-controller	error-404-isvc-31e6a-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-31e6a-predictor-764bc8c74b from 0 to 1
21:11:09	kserve-ci-e2e-test	kubelet	error-404-isvc-46851-predictor-7f76877c8-jkqxj	Killing	Stopping container kserve-container
21:11:10	kserve-ci-e2e-test	kubelet	success-200-isvc-31e6a-predictor-6c64485579-ztxgg	Created	Created container: kube-rbac-proxy
21:11:10	kserve-ci-e2e-test	multus	success-200-isvc-31e6a-predictor-6c64485579-ztxgg	AddedInterface	Add eth0 [10.134.0.40/23] from ovn-kubernetes
21:11:10	kserve-ci-e2e-test	kubelet	success-200-isvc-31e6a-predictor-6c64485579-ztxgg	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1999" already present on machine
21:11:10	kserve-ci-e2e-test	kubelet	success-200-isvc-31e6a-predictor-6c64485579-ztxgg	Created	Created container: kserve-container
21:11:10 (x2)	kserve-ci-e2e-test	kubelet	error-404-isvc-31e6a-predictor-764bc8c74b-kd5vf	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "error-404-isvc-31e6a-predictor-serving-cert" not found
21:11:10	kserve-ci-e2e-test	kubelet	success-200-isvc-31e6a-predictor-6c64485579-ztxgg	Started	Started container kube-rbac-proxy
21:11:10	kserve-ci-e2e-test	kubelet	success-200-isvc-31e6a-predictor-6c64485579-ztxgg	Started	Started container kserve-container
21:11:10	kserve-ci-e2e-test	kubelet	success-200-isvc-31e6a-predictor-6c64485579-ztxgg	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:11:10	kserve-ci-e2e-test	kubelet	success-200-isvc-46851-predictor-5c8c955cdc-c8zks	Unhealthy	Readiness probe failed: Get "https://10.134.0.37:8643/healthz": dial tcp 10.134.0.37:8643: connect: connection refused
21:11:11	kserve-ci-e2e-test	kubelet	error-404-isvc-31e6a-predictor-764bc8c74b-kd5vf	Started	Started container kube-rbac-proxy
21:11:11	kserve-ci-e2e-test	kubelet	error-404-isvc-31e6a-predictor-764bc8c74b-kd5vf	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:11:11	kserve-ci-e2e-test	multus	error-404-isvc-31e6a-predictor-764bc8c74b-kd5vf	AddedInterface	Add eth0 [10.133.0.24/23] from ovn-kubernetes
21:11:11	kserve-ci-e2e-test	kubelet	error-404-isvc-31e6a-predictor-764bc8c74b-kd5vf	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1999" already present on machine
21:11:11	kserve-ci-e2e-test	kubelet	error-404-isvc-31e6a-predictor-764bc8c74b-kd5vf	Created	Created container: kserve-container
21:11:11	kserve-ci-e2e-test	kubelet	error-404-isvc-31e6a-predictor-764bc8c74b-kd5vf	Created	Created container: kube-rbac-proxy
21:11:11	kserve-ci-e2e-test	kubelet	error-404-isvc-46851-predictor-7f76877c8-jkqxj	Unhealthy	Readiness probe failed: Get "https://10.133.0.22:8643/healthz": dial tcp 10.133.0.22:8643: connect: connection refused
21:11:11 (x6)	kserve-ci-e2e-test	kubelet	sequence-graph-ed647-5445897977-p9svl	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
21:11:11	kserve-ci-e2e-test	kubelet	error-404-isvc-31e6a-predictor-764bc8c74b-kd5vf	Started	Started container kserve-container
21:11:30 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-62e6f-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:11:30 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-62e6f-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:11:34 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-62e6f-predictor-6984c9779b-6qqq9	Unhealthy	Readiness probe failed: dial tcp 10.133.0.23:8080: connect: connection refused
21:11:35 (x7)	kserve-ci-e2e-test	kubelet	success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r	Unhealthy	Readiness probe failed: dial tcp 10.134.0.38:8080: connect: connection refused
21:11:36 (x6)	kserve-ci-e2e-test	kubelet	ensemble-graph-46851-555c4ddb89-n7mwm	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
21:11:39 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-31e6a-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:11:39 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-31e6a-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:11:44	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-62e6f	InferenceServiceReady	InferenceService [error-404-isvc-62e6f] is Ready
21:11:44 (x5)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-62e6f	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:11:45	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-62e6f	InferenceServiceReady	InferenceService [success-200-isvc-62e6f] is Ready
21:11:45 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-62e6f	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:11:54 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-31e6a-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:11:54 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-31e6a-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:11:55	kserve-ci-e2e-test	InferenceGraphController	sequence-graph-62e6f	InferenceGraphReady	InferenceGraph [sequence-graph-62e6f] is Ready
21:11:55	kserve-ci-e2e-test	replicaset-controller	sequence-graph-62e6f-7c8567c89d	SuccessfulCreate	Created pod: sequence-graph-62e6f-7c8567c89d-p74m9
21:11:55	kserve-ci-e2e-test	kubelet	sequence-graph-62e6f-7c8567c89d-p74m9	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "sequence-graph-62e6f-serving-cert" not found
21:11:55	kserve-ci-e2e-test	deployment-controller	sequence-graph-62e6f	ScalingReplicaSet	Scaled up replica set sequence-graph-62e6f-7c8567c89d from 0 to 1
21:11:56	kserve-ci-e2e-test	multus	sequence-graph-62e6f-7c8567c89d-p74m9	AddedInterface	Add eth0 [10.134.0.41/23] from ovn-kubernetes
21:11:56	kserve-ci-e2e-test	kubelet	sequence-graph-62e6f-7c8567c89d-p74m9	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:2c5833579293a5043b5db03d2dc3bee6dec8d8c7a284a9a790353f02c7a72dd3" already present on machine
21:11:57	kserve-ci-e2e-test	kubelet	sequence-graph-62e6f-7c8567c89d-p74m9	Created	Created container: sequence-graph-62e6f
21:11:57	kserve-ci-e2e-test	kubelet	sequence-graph-62e6f-7c8567c89d-p74m9	Started	Started container sequence-graph-62e6f
21:11:59 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-31e6a-predictor-764bc8c74b-kd5vf	Unhealthy	Readiness probe failed: dial tcp 10.133.0.24:8080: connect: connection refused
21:12:00 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-62e6f-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:12:00 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-62e6f-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:12:00 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-62e6f-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:12:00 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-62e6f-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:12:06	kserve-ci-e2e-test	kubelet	sequence-graph-62e6f-7c8567c89d-p74m9	Killing	Stopping container sequence-graph-62e6f
21:12:07	kserve-ci-e2e-test	kubelet	success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r	Killing	Stopping container kserve-container
21:12:07	kserve-ci-e2e-test	multus	error-404-isvc-b5a48-predictor-5bb6b4f9f8-4b79b	AddedInterface	Add eth0 [10.133.0.25/23] from ovn-kubernetes
21:12:07	kserve-ci-e2e-test	deployment-controller	success-200-isvc-b5a48-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-b5a48-predictor-86f5cd75f from 0 to 1
21:12:07	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-b5a48-predictor-5bb6b4f9f8	SuccessfulCreate	Created pod: error-404-isvc-b5a48-predictor-5bb6b4f9f8-4b79b
21:12:07	kserve-ci-e2e-test	deployment-controller	error-404-isvc-b5a48-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-b5a48-predictor-5bb6b4f9f8 from 0 to 1
21:12:07	kserve-ci-e2e-test	kubelet	error-404-isvc-62e6f-predictor-6984c9779b-6qqq9	Killing	Stopping container kserve-container
21:12:07	kserve-ci-e2e-test	kubelet	success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd	Started	Started container kube-rbac-proxy
21:12:07	kserve-ci-e2e-test	kubelet	success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r	Killing	Stopping container kube-rbac-proxy
21:12:07	kserve-ci-e2e-test	kubelet	error-404-isvc-62e6f-predictor-6984c9779b-6qqq9	Killing	Stopping container kube-rbac-proxy
21:12:07	kserve-ci-e2e-test	kubelet	error-404-isvc-b5a48-predictor-5bb6b4f9f8-4b79b	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1999" already present on machine
21:12:07	kserve-ci-e2e-test	kubelet	success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd	Created	Created container: kube-rbac-proxy
21:12:07	kserve-ci-e2e-test	kubelet	success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:12:07	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-b5a48-predictor-86f5cd75f	SuccessfulCreate	Created pod: success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd
21:12:07	kserve-ci-e2e-test	kubelet	success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd	Started	Started container kserve-container
21:12:07	kserve-ci-e2e-test	kubelet	success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd	Created	Created container: kserve-container
21:12:07	kserve-ci-e2e-test	kubelet	success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1999" already present on machine
21:12:07	kserve-ci-e2e-test	multus	success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd	AddedInterface	Add eth0 [10.134.0.42/23] from ovn-kubernetes
21:12:07	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-b5a48	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-b5a48": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-b5a48": the object has been modified; please apply your changes to the latest version and try again
21:12:07	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-b5a48	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-b5a48": the object has been modified; please apply your changes to the latest version and try again
21:12:08	kserve-ci-e2e-test	kubelet	error-404-isvc-b5a48-predictor-5bb6b4f9f8-4b79b	Created	Created container: kube-rbac-proxy
21:12:08	kserve-ci-e2e-test	kubelet	error-404-isvc-b5a48-predictor-5bb6b4f9f8-4b79b	Started	Started container kserve-container
21:12:08 (x6)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-31e6a	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:12:08	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-31e6a	InferenceServiceReady	InferenceService [success-200-isvc-31e6a] is Ready
21:12:08	kserve-ci-e2e-test	kubelet	error-404-isvc-b5a48-predictor-5bb6b4f9f8-4b79b	Created	Created container: kserve-container
21:12:08	kserve-ci-e2e-test	kubelet	error-404-isvc-b5a48-predictor-5bb6b4f9f8-4b79b	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:12:08	kserve-ci-e2e-test	kubelet	error-404-isvc-b5a48-predictor-5bb6b4f9f8-4b79b	Started	Started container kube-rbac-proxy
21:12:09	kserve-ci-e2e-test	kubelet	error-404-isvc-62e6f-predictor-6984c9779b-6qqq9	Unhealthy	Readiness probe failed: Get "https://10.133.0.23:8643/healthz": dial tcp 10.133.0.23:8643: connect: connection refused
21:12:09 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-31e6a	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:12:09	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-31e6a	InferenceServiceReady	InferenceService [error-404-isvc-31e6a] is Ready
21:12:10	kserve-ci-e2e-test	kubelet	success-200-isvc-62e6f-predictor-545f99f9cc-x7b7r	Unhealthy	Readiness probe failed: Get "https://10.134.0.38:8643/healthz": dial tcp 10.134.0.38:8643: connect: connection refused
21:12:19	kserve-ci-e2e-test	InferenceGraphController	ensemble-graph-31e6a	InferenceGraphReady	InferenceGraph [ensemble-graph-31e6a] is Ready
21:12:19	kserve-ci-e2e-test	replicaset-controller	ensemble-graph-31e6a-b7ddd645d	SuccessfulCreate	Created pod: ensemble-graph-31e6a-b7ddd645d-m4f2b
21:12:19	kserve-ci-e2e-test	deployment-controller	ensemble-graph-31e6a	ScalingReplicaSet	Scaled up replica set ensemble-graph-31e6a-b7ddd645d from 0 to 1
21:12:19	kserve-ci-e2e-test	kubelet	ensemble-graph-31e6a-b7ddd645d-m4f2b	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "ensemble-graph-31e6a-serving-cert" not found
21:12:20	kserve-ci-e2e-test	kubelet	ensemble-graph-31e6a-b7ddd645d-m4f2b	Created	Created container: ensemble-graph-31e6a
21:12:20	kserve-ci-e2e-test	kubelet	ensemble-graph-31e6a-b7ddd645d-m4f2b	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:2c5833579293a5043b5db03d2dc3bee6dec8d8c7a284a9a790353f02c7a72dd3" already present on machine
21:12:20	kserve-ci-e2e-test	kubelet	ensemble-graph-31e6a-b7ddd645d-m4f2b	Started	Started container ensemble-graph-31e6a
21:12:20	kserve-ci-e2e-test	multus	ensemble-graph-31e6a-b7ddd645d-m4f2b	AddedInterface	Add eth0 [10.134.0.43/23] from ovn-kubernetes
21:12:24 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-31e6a-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:12:24 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-31e6a-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:12:33 (x6)	kserve-ci-e2e-test	kubelet	sequence-graph-62e6f-7c8567c89d-p74m9	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
21:12:39 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-31e6a-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:12:39 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-31e6a-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:13:04 (x6)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-b5a48	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:13:04	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-b5a48	InferenceServiceReady	InferenceService [success-200-isvc-b5a48] is Ready
21:13:04 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	ensemble-graph-31e6a	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:13:04 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	ensemble-graph-31e6a	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:13:05 (x5)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-b5a48	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:13:05	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-b5a48	InferenceServiceReady	InferenceService [error-404-isvc-b5a48] is Ready
21:13:17	kserve-ci-e2e-test	InferenceGraphController	sequence-graph-b5a48	InferenceGraphReady	InferenceGraph [sequence-graph-b5a48] is Ready
21:13:17	kserve-ci-e2e-test	kubelet	sequence-graph-b5a48-848c5fd7d8-nzfp4	Started	Started container sequence-graph-b5a48
21:13:17	kserve-ci-e2e-test	kubelet	sequence-graph-b5a48-848c5fd7d8-nzfp4	Created	Created container: sequence-graph-b5a48
21:13:17	kserve-ci-e2e-test	multus	sequence-graph-b5a48-848c5fd7d8-nzfp4	AddedInterface	Add eth0 [10.134.0.44/23] from ovn-kubernetes
21:13:17	kserve-ci-e2e-test	kubelet	sequence-graph-b5a48-848c5fd7d8-nzfp4	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:2c5833579293a5043b5db03d2dc3bee6dec8d8c7a284a9a790353f02c7a72dd3" already present on machine
21:13:17	kserve-ci-e2e-test	replicaset-controller	sequence-graph-b5a48-848c5fd7d8	SuccessfulCreate	Created pod: sequence-graph-b5a48-848c5fd7d8-nzfp4
21:13:17	kserve-ci-e2e-test	deployment-controller	sequence-graph-b5a48	ScalingReplicaSet	Scaled up replica set sequence-graph-b5a48-848c5fd7d8 from 0 to 1
21:13:22 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-b5a48-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:13:22 (x5)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-b5a48-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:13:32	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-b5a48	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:13:32	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-b5a48	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:13:37 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-b5a48-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:13:37 (x6)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-b5a48-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:13:47	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-b5a48	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:13:47	kserve-ci-e2e-test	horizontal-pod-autoscaler	sequence-graph-b5a48	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:20:34	kserve-ci-e2e-test	kubelet	success-200-isvc-31e6a-predictor-6c64485579-ztxgg	Killing	Stopping container kserve-container
21:20:34	kserve-ci-e2e-test	kubelet	success-200-isvc-31e6a-predictor-6c64485579-ztxgg	Killing	Stopping container kube-rbac-proxy
21:20:34	kserve-ci-e2e-test	kubelet	ensemble-graph-31e6a-b7ddd645d-m4f2b	Killing	Stopping container ensemble-graph-31e6a
21:20:35	kserve-ci-e2e-test	deployment-controller	success-200-isvc-d3600-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-d3600-predictor-7d54b7f68f from 0 to 1
21:20:35	kserve-ci-e2e-test	deployment-controller	error-404-isvc-d3600-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-d3600-predictor-5bc9bb5464 from 0 to 1
21:20:35	kserve-ci-e2e-test	kubelet	error-404-isvc-31e6a-predictor-764bc8c74b-kd5vf	Killing	Stopping container kube-rbac-proxy
21:20:35	kserve-ci-e2e-test	kubelet	error-404-isvc-31e6a-predictor-764bc8c74b-kd5vf	Killing	Stopping container kserve-container
21:20:35	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-d3600-predictor-7d54b7f68f	SuccessfulCreate	Created pod: success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw
21:20:35 (x2)	kserve-ci-e2e-test	kubelet	success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "success-200-isvc-d3600-predictor-serving-cert" not found
21:20:35	kserve-ci-e2e-test	multus	error-404-isvc-d3600-predictor-5bc9bb5464-rns9p	AddedInterface	Add eth0 [10.133.0.26/23] from ovn-kubernetes
21:20:35	kserve-ci-e2e-test	kubelet	error-404-isvc-d3600-predictor-5bc9bb5464-rns9p	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1999" already present on machine
21:20:35	kserve-ci-e2e-test	kubelet	error-404-isvc-d3600-predictor-5bc9bb5464-rns9p	Created	Created container: kserve-container
21:20:35	kserve-ci-e2e-test	kubelet	error-404-isvc-d3600-predictor-5bc9bb5464-rns9p	Started	Started container kserve-container
21:20:35	kserve-ci-e2e-test	kubelet	error-404-isvc-d3600-predictor-5bc9bb5464-rns9p	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:20:35	kserve-ci-e2e-test	kubelet	error-404-isvc-d3600-predictor-5bc9bb5464-rns9p	Created	Created container: kube-rbac-proxy
21:20:35	kserve-ci-e2e-test	kubelet	error-404-isvc-d3600-predictor-5bc9bb5464-rns9p	Started	Started container kube-rbac-proxy
21:20:35	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-d3600-predictor-5bc9bb5464	SuccessfulCreate	Created pod: error-404-isvc-d3600-predictor-5bc9bb5464-rns9p
21:20:37	kserve-ci-e2e-test	kubelet	success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw	Created	Created container: kube-rbac-proxy
21:20:37	kserve-ci-e2e-test	kubelet	success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw	Started	Started container kube-rbac-proxy
21:20:37	kserve-ci-e2e-test	kubelet	success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw	Started	Started container kserve-container
21:20:37	kserve-ci-e2e-test	multus	success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw	AddedInterface	Add eth0 [10.134.0.45/23] from ovn-kubernetes
21:20:37	kserve-ci-e2e-test	kubelet	success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1999" already present on machine
21:20:37	kserve-ci-e2e-test	kubelet	success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw	Created	Created container: kserve-container
21:20:37	kserve-ci-e2e-test	kubelet	success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:20:38	kserve-ci-e2e-test	kubelet	success-200-isvc-31e6a-predictor-6c64485579-ztxgg	Unhealthy	Readiness probe failed: Get "https://10.134.0.40:8643/healthz": dial tcp 10.134.0.40:8643: connect: connection refused
21:20:38 (x8)	kserve-ci-e2e-test	kubelet	success-200-isvc-31e6a-predictor-6c64485579-ztxgg	Unhealthy	Readiness probe failed: dial tcp 10.134.0.40:8080: connect: connection refused
21:20:50	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-d3600-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:20:50	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-d3600-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:21:02 (x6)	kserve-ci-e2e-test	kubelet	ensemble-graph-31e6a-b7ddd645d-m4f2b	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
21:21:20 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-d3600-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:21:20 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-d3600-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:21:23 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-d3600-predictor-5bc9bb5464-rns9p	Unhealthy	Readiness probe failed: dial tcp 10.133.0.26:8080: connect: connection refused
21:21:24 (x7)	kserve-ci-e2e-test	kubelet	success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw	Unhealthy	Readiness probe failed: dial tcp 10.134.0.45:8080: connect: connection refused
21:21:32	kserve-ci-e2e-test	kubelet	error-404-isvc-b5a48-predictor-5bb6b4f9f8-4b79b	Killing	Stopping container kube-rbac-proxy
21:21:32	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-3ccd0-predictor-5778dff7cc	SuccessfulCreate	Created pod: success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg
21:21:32	kserve-ci-e2e-test	kubelet	sequence-graph-b5a48-848c5fd7d8-nzfp4	Killing	Stopping container sequence-graph-b5a48
21:21:32	kserve-ci-e2e-test	kubelet	error-404-isvc-b5a48-predictor-5bb6b4f9f8-4b79b	Killing	Stopping container kserve-container
21:21:32	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-3ccd0-predictor-5c789f8bd8	SuccessfulCreate	Created pod: error-404-isvc-3ccd0-predictor-5c789f8bd8-c7nfl
21:21:32	kserve-ci-e2e-test	kubelet	error-404-isvc-3ccd0-predictor-5c789f8bd8-c7nfl	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "error-404-isvc-3ccd0-predictor-serving-cert" not found
21:21:32	kserve-ci-e2e-test	deployment-controller	error-404-isvc-3ccd0-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-3ccd0-predictor-5c789f8bd8 from 0 to 1
21:21:32	kserve-ci-e2e-test	kubelet	success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd	Killing	Stopping container kube-rbac-proxy
21:21:32	kserve-ci-e2e-test	deployment-controller	success-200-isvc-3ccd0-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-3ccd0-predictor-5778dff7cc from 0 to 1
21:21:32	kserve-ci-e2e-test	kubelet	success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd	Killing	Stopping container kserve-container
21:21:33	kserve-ci-e2e-test	kubelet	success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1999" already present on machine
21:21:33	kserve-ci-e2e-test	kubelet	success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:21:33	kserve-ci-e2e-test	multus	error-404-isvc-3ccd0-predictor-5c789f8bd8-c7nfl	AddedInterface	Add eth0 [10.133.0.27/23] from ovn-kubernetes
21:21:33	kserve-ci-e2e-test	kubelet	error-404-isvc-3ccd0-predictor-5c789f8bd8-c7nfl	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1999" already present on machine
21:21:33	kserve-ci-e2e-test	kubelet	success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg	Started	Started container kube-rbac-proxy
21:21:33	kserve-ci-e2e-test	kubelet	success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg	Created	Created container: kube-rbac-proxy
21:21:33	kserve-ci-e2e-test	multus	success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg	AddedInterface	Add eth0 [10.134.0.46/23] from ovn-kubernetes
21:21:33	kserve-ci-e2e-test	kubelet	success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg	Created	Created container: kserve-container
21:21:33	kserve-ci-e2e-test	kubelet	success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg	Started	Started container kserve-container
21:21:33	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-d3600	InferenceServiceReady	InferenceService [error-404-isvc-d3600] is Ready
21:21:33 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-d3600	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:21:33	kserve-ci-e2e-test	kubelet	error-404-isvc-3ccd0-predictor-5c789f8bd8-c7nfl	Started	Started container kube-rbac-proxy
21:21:33	kserve-ci-e2e-test	kubelet	error-404-isvc-3ccd0-predictor-5c789f8bd8-c7nfl	Created	Created container: kube-rbac-proxy
21:21:33	kserve-ci-e2e-test	kubelet	error-404-isvc-3ccd0-predictor-5c789f8bd8-c7nfl	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:21:33	kserve-ci-e2e-test	kubelet	error-404-isvc-3ccd0-predictor-5c789f8bd8-c7nfl	Started	Started container kserve-container
21:21:33	kserve-ci-e2e-test	kubelet	error-404-isvc-3ccd0-predictor-5c789f8bd8-c7nfl	Created	Created container: kserve-container
21:21:34 (x8)	kserve-ci-e2e-test	kubelet	success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd	Unhealthy	Readiness probe failed: dial tcp 10.134.0.42:8080: connect: connection refused
21:21:34	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-d3600	InferenceServiceReady	InferenceService [success-200-isvc-d3600] is Ready
21:21:34 (x6)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-d3600	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:21:34	kserve-ci-e2e-test	kubelet	success-200-isvc-b5a48-predictor-86f5cd75f-c8pqd	Unhealthy	Readiness probe failed: Get "https://10.134.0.42:8643/healthz": dial tcp 10.134.0.42:8643: connect: connection refused
21:21:35	kserve-ci-e2e-test	kubelet	error-404-isvc-b5a48-predictor-5bb6b4f9f8-4b79b	Unhealthy	Readiness probe failed: Get "https://10.133.0.25:8643/healthz": dial tcp 10.133.0.25:8643: connect: connection refused
21:21:35 (x8)	kserve-ci-e2e-test	kubelet	error-404-isvc-b5a48-predictor-5bb6b4f9f8-4b79b	Unhealthy	Readiness probe failed: dial tcp 10.133.0.25:8080: connect: connection refused
21:21:45	kserve-ci-e2e-test	deployment-controller	splitter-graph-d3600	ScalingReplicaSet	Scaled up replica set splitter-graph-d3600-78645b4dcf from 0 to 1
21:21:45	kserve-ci-e2e-test	InferenceGraphController	splitter-graph-d3600	InferenceGraphReady	InferenceGraph [splitter-graph-d3600] is Ready
21:21:45	kserve-ci-e2e-test	replicaset-controller	splitter-graph-d3600-78645b4dcf	SuccessfulCreate	Created pod: splitter-graph-d3600-78645b4dcf-b5xv7
21:21:45	kserve-ci-e2e-test	kubelet	splitter-graph-d3600-78645b4dcf-b5xv7	Started	Started container splitter-graph-d3600
21:21:45	kserve-ci-e2e-test	kubelet	splitter-graph-d3600-78645b4dcf-b5xv7	Created	Created container: splitter-graph-d3600
21:21:45	kserve-ci-e2e-test	kubelet	splitter-graph-d3600-78645b4dcf-b5xv7	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:2c5833579293a5043b5db03d2dc3bee6dec8d8c7a284a9a790353f02c7a72dd3" already present on machine
21:21:45	kserve-ci-e2e-test	multus	splitter-graph-d3600-78645b4dcf-b5xv7	AddedInterface	Add eth0 [10.134.0.47/23] from ovn-kubernetes
21:21:50 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-d3600-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:21:50 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-d3600-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:21:50 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-d3600-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:21:50 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-d3600-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:21:56	kserve-ci-e2e-test	kubelet	splitter-graph-d3600-78645b4dcf-b5xv7	Killing	Stopping container splitter-graph-d3600
21:21:56	kserve-ci-e2e-test	kubelet	success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw	Killing	Stopping container kserve-container
21:21:56	kserve-ci-e2e-test	kubelet	error-404-isvc-d3600-predictor-5bc9bb5464-rns9p	Killing	Stopping container kube-rbac-proxy
21:21:56	kserve-ci-e2e-test	kubelet	error-404-isvc-a1ff2-predictor-6bd48997f4-pp8xj	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "error-404-isvc-a1ff2-predictor-serving-cert" not found
21:21:56	kserve-ci-e2e-test	replicaset-controller	success-200-isvc-a1ff2-predictor-794d58c9dc	SuccessfulCreate	Created pod: success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v
21:21:56	kserve-ci-e2e-test	deployment-controller	success-200-isvc-a1ff2-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-a1ff2-predictor-794d58c9dc from 0 to 1
21:21:56	kserve-ci-e2e-test	kubelet	error-404-isvc-d3600-predictor-5bc9bb5464-rns9p	Killing	Stopping container kserve-container
21:21:56	kserve-ci-e2e-test	deployment-controller	error-404-isvc-a1ff2-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-a1ff2-predictor-6bd48997f4 from 0 to 1
21:21:56	kserve-ci-e2e-test	kubelet	success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "success-200-isvc-a1ff2-predictor-serving-cert" not found
21:21:56	kserve-ci-e2e-test	replicaset-controller	error-404-isvc-a1ff2-predictor-6bd48997f4	SuccessfulCreate	Created pod: error-404-isvc-a1ff2-predictor-6bd48997f4-pp8xj
21:21:56	kserve-ci-e2e-test	kubelet	success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw	Killing	Stopping container kube-rbac-proxy
21:21:57	kserve-ci-e2e-test	kubelet	success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:21:57	kserve-ci-e2e-test	multus	error-404-isvc-a1ff2-predictor-6bd48997f4-pp8xj	AddedInterface	Add eth0 [10.133.0.28/23] from ovn-kubernetes
21:21:57	kserve-ci-e2e-test	kubelet	error-404-isvc-a1ff2-predictor-6bd48997f4-pp8xj	Started	Started container kube-rbac-proxy
21:21:57	kserve-ci-e2e-test	kubelet	success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v	Started	Started container kserve-container
21:21:57	kserve-ci-e2e-test	kubelet	error-404-isvc-a1ff2-predictor-6bd48997f4-pp8xj	Pulled	Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine
21:21:57	kserve-ci-e2e-test	kubelet	success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v	Created	Created container: kserve-container
21:21:57	kserve-ci-e2e-test	kubelet	error-404-isvc-a1ff2-predictor-6bd48997f4-pp8xj	Started	Started container kserve-container
21:21:57	kserve-ci-e2e-test	kubelet	success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1999" already present on machine
21:21:57	kserve-ci-e2e-test	kubelet	success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v	Started	Started container kube-rbac-proxy
21:21:57	kserve-ci-e2e-test	kubelet	success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v	Created	Created container: kube-rbac-proxy
21:21:57	kserve-ci-e2e-test	multus	success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v	AddedInterface	Add eth0 [10.134.0.48/23] from ovn-kubernetes
21:21:57	kserve-ci-e2e-test	kubelet	error-404-isvc-a1ff2-predictor-6bd48997f4-pp8xj	Created	Created container: kube-rbac-proxy
21:21:57	kserve-ci-e2e-test	kubelet	error-404-isvc-a1ff2-predictor-6bd48997f4-pp8xj	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1999" already present on machine
21:21:57	kserve-ci-e2e-test	kubelet	error-404-isvc-a1ff2-predictor-6bd48997f4-pp8xj	Created	Created container: kserve-container
21:21:58	kserve-ci-e2e-test	kubelet	error-404-isvc-d3600-predictor-5bc9bb5464-rns9p	Unhealthy	Readiness probe failed: Get "https://10.133.0.26:8643/healthz": dial tcp 10.133.0.26:8643: connect: connection refused
21:21:59	kserve-ci-e2e-test	kubelet	success-200-isvc-d3600-predictor-7d54b7f68f-2s7fw	Unhealthy	Readiness probe failed: Get "https://10.134.0.45:8643/healthz": dial tcp 10.134.0.45:8643: connect: connection refused
21:21:59 (x6)	kserve-ci-e2e-test	kubelet	sequence-graph-b5a48-848c5fd7d8-nzfp4	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
21:22:02 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-3ccd0-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:22:02 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-3ccd0-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:22:11	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-a1ff2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:22:11	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-a1ff2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:22:17 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-3ccd0-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:22:17 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-3ccd0-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:22:20 (x7)	kserve-ci-e2e-test	kubelet	success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg	Unhealthy	Readiness probe failed: dial tcp 10.134.0.46:8080: connect: connection refused
21:22:20 (x7)	kserve-ci-e2e-test	kubelet	error-404-isvc-3ccd0-predictor-5c789f8bd8-c7nfl	Unhealthy	Readiness probe failed: dial tcp 10.133.0.27:8080: connect: connection refused
21:22:22 (x6)	kserve-ci-e2e-test	kubelet	splitter-graph-d3600-78645b4dcf-b5xv7	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
21:22:26 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-a1ff2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:22:26 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-a1ff2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:22:30	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-3ccd0	InferenceServiceReady	InferenceService [error-404-isvc-3ccd0] is Ready
21:22:30 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-3ccd0	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:22:30 (x5)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-3ccd0	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:22:30	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-3ccd0	InferenceServiceReady	InferenceService [success-200-isvc-3ccd0] is Ready
21:22:42	kserve-ci-e2e-test	deployment-controller	switch-graph-3ccd0	ScalingReplicaSet	Scaled up replica set switch-graph-3ccd0-6464d894c from 0 to 1
21:22:42	kserve-ci-e2e-test	InferenceGraphController	switch-graph-3ccd0	InferenceGraphReady	InferenceGraph [switch-graph-3ccd0] is Ready
21:22:42	kserve-ci-e2e-test	replicaset-controller	switch-graph-3ccd0-6464d894c	SuccessfulCreate	Created pod: switch-graph-3ccd0-6464d894c-mplj9
21:22:43	kserve-ci-e2e-test	kubelet	switch-graph-3ccd0-6464d894c-mplj9	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:2c5833579293a5043b5db03d2dc3bee6dec8d8c7a284a9a790353f02c7a72dd3" already present on machine
21:22:43	kserve-ci-e2e-test	kubelet	switch-graph-3ccd0-6464d894c-mplj9	Started	Started container switch-graph-3ccd0
21:22:43	kserve-ci-e2e-test	kubelet	switch-graph-3ccd0-6464d894c-mplj9	Created	Created container: switch-graph-3ccd0
21:22:43	kserve-ci-e2e-test	multus	switch-graph-3ccd0-6464d894c-mplj9	AddedInterface	Add eth0 [10.134.0.49/23] from ovn-kubernetes
21:22:54 (x6)	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-a1ff2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:22:54	kserve-ci-e2e-test	v1beta1Controllers	success-200-isvc-a1ff2	InferenceServiceReady	InferenceService [success-200-isvc-a1ff2] is Ready
21:22:55	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-a1ff2	InferenceServiceReady	InferenceService [error-404-isvc-a1ff2] is Ready
21:22:55 (x6)	kserve-ci-e2e-test	v1beta1Controllers	error-404-isvc-a1ff2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
21:22:57	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-3ccd0	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:22:57	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-3ccd0	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:23:02 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-3ccd0-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:23:02 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-3ccd0-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:23:02 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-3ccd0-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:23:02 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-3ccd0-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:23:06	kserve-ci-e2e-test	kubelet	splitter-graph-a1ff2-574bdd87b5-hktnd	FailedMount	MountVolume.SetUp failed for volume "proxy-tls" : secret "splitter-graph-a1ff2-serving-cert" not found
21:23:06	kserve-ci-e2e-test	deployment-controller	splitter-graph-a1ff2	ScalingReplicaSet	Scaled up replica set splitter-graph-a1ff2-574bdd87b5 from 0 to 1
21:23:06	kserve-ci-e2e-test	replicaset-controller	splitter-graph-a1ff2-574bdd87b5	SuccessfulCreate	Created pod: splitter-graph-a1ff2-574bdd87b5-hktnd
21:23:06	kserve-ci-e2e-test	InferenceGraphController	splitter-graph-a1ff2	InferenceGraphReady	InferenceGraph [splitter-graph-a1ff2] is Ready
21:23:07	kserve-ci-e2e-test	kubelet	splitter-graph-a1ff2-574bdd87b5-hktnd	Started	Started container splitter-graph-a1ff2
21:23:07	kserve-ci-e2e-test	multus	splitter-graph-a1ff2-574bdd87b5-hktnd	AddedInterface	Add eth0 [10.134.0.50/23] from ovn-kubernetes
21:23:07	kserve-ci-e2e-test	kubelet	splitter-graph-a1ff2-574bdd87b5-hktnd	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:2c5833579293a5043b5db03d2dc3bee6dec8d8c7a284a9a790353f02c7a72dd3" already present on machine
21:23:07	kserve-ci-e2e-test	kubelet	splitter-graph-a1ff2-574bdd87b5-hktnd	Created	Created container: splitter-graph-a1ff2
21:23:11 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-a1ff2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:23:11 (x3)	kserve-ci-e2e-test	horizontal-pod-autoscaler	error-404-isvc-a1ff2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:23:11 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-a1ff2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:23:11 (x4)	kserve-ci-e2e-test	horizontal-pod-autoscaler	success-200-isvc-a1ff2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:23:12	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-3ccd0	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:23:12	kserve-ci-e2e-test	horizontal-pod-autoscaler	switch-graph-3ccd0	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
21:23:36 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	splitter-graph-a1ff2	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:23:36 (x2)	kserve-ci-e2e-test	horizontal-pod-autoscaler	splitter-graph-a1ff2	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
21:31:21	kserve-ci-e2e-test	kubelet	success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v	Killing	Stopping container kserve-container
21:31:21	kserve-ci-e2e-test	kubelet	success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v	Killing	Stopping container kube-rbac-proxy
21:31:21	kserve-ci-e2e-test	kubelet	error-404-isvc-a1ff2-predictor-6bd48997f4-pp8xj	Killing	Stopping container kube-rbac-proxy
21:31:21	kserve-ci-e2e-test	kubelet	error-404-isvc-a1ff2-predictor-6bd48997f4-pp8xj	Killing	Stopping container kserve-container
21:31:21	kserve-ci-e2e-test	kubelet	splitter-graph-a1ff2-574bdd87b5-hktnd	Killing	Stopping container splitter-graph-a1ff2
21:31:24 (x8)	kserve-ci-e2e-test	kubelet	error-404-isvc-a1ff2-predictor-6bd48997f4-pp8xj	Unhealthy	Readiness probe failed: dial tcp 10.133.0.28:8080: connect: connection refused
21:31:24	kserve-ci-e2e-test	kubelet	success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v	Unhealthy	Readiness probe failed: Get "https://10.134.0.48:8643/healthz": dial tcp 10.134.0.48:8643: connect: connection refused
21:31:24 (x8)	kserve-ci-e2e-test	kubelet	success-200-isvc-a1ff2-predictor-794d58c9dc-jzd8v	Unhealthy	Readiness probe failed: dial tcp 10.134.0.48:8080: connect: connection refused
21:31:24	kserve-ci-e2e-test	kubelet	error-404-isvc-a1ff2-predictor-6bd48997f4-pp8xj	Unhealthy	Readiness probe failed: Get "https://10.133.0.28:8643/healthz": dial tcp 10.133.0.28:8643: connect: connection refused
21:31:49 (x6)	kserve-ci-e2e-test	kubelet	splitter-graph-a1ff2-574bdd87b5-hktnd	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503
21:39:02	kserve-ci-e2e-test	kubelet	error-404-isvc-3ccd0-predictor-5c789f8bd8-c7nfl	Killing	Stopping container kserve-container
21:39:02	kserve-ci-e2e-test	kubelet	error-404-isvc-3ccd0-predictor-5c789f8bd8-c7nfl	Killing	Stopping container kube-rbac-proxy
21:39:02	kserve-ci-e2e-test	kubelet	switch-graph-3ccd0-6464d894c-mplj9	Killing	Stopping container switch-graph-3ccd0
21:39:02	kserve-ci-e2e-test	kubelet	success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg	Killing	Stopping container kserve-container
21:39:02	kserve-ci-e2e-test	kubelet	success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg	Killing	Stopping container kube-rbac-proxy
21:39:05	kserve-ci-e2e-test	kubelet	error-404-isvc-3ccd0-predictor-5c789f8bd8-c7nfl	Unhealthy	Readiness probe failed: Get "https://10.133.0.27:8643/healthz": dial tcp 10.133.0.27:8643: connect: connection refused
21:39:05	kserve-ci-e2e-test	kubelet	success-200-isvc-3ccd0-predictor-5778dff7cc-kz9kg	Unhealthy	Readiness probe failed: Get "https://10.134.0.46:8643/healthz": dial tcp 10.134.0.46:8643: connect: connection refused
21:39:29 (x6)	kserve-ci-e2e-test	kubelet	switch-graph-3ccd0-6464d894c-mplj9	Unhealthy	Readiness probe failed: HTTP probe failed with statuscode: 503