Events

Time	Namespace	Component	RelatedObject	Reason	Message
22:03:02	e2e-gw0-a3ce8b9d69		isvc-xgboost-graph-predictor-74db998dbd-gv7p2	Scheduled	Successfully assigned e2e-gw0-a3ce8b9d69/isvc-xgboost-graph-predictor-74db998dbd-gv7p2 to ip-10-0-137-4.us-east-2.compute.internal
22:15:13	e2e-gw0-a3ce8b9d69		success-200-isvc-47126-predictor-77fcdcc8cb-zwn9w	Scheduled	Successfully assigned e2e-gw0-a3ce8b9d69/success-200-isvc-47126-predictor-77fcdcc8cb-zwn9w to ip-10-0-140-139.us-east-2.compute.internal
22:26:11	e2e-gw0-a3ce8b9d69		error-404-isvc-18df3-predictor-7b5568b5d9-k48m5	Scheduled	Successfully assigned e2e-gw0-a3ce8b9d69/error-404-isvc-18df3-predictor-7b5568b5d9-k48m5 to ip-10-0-137-4.us-east-2.compute.internal
22:15:13	e2e-gw0-a3ce8b9d69		error-404-isvc-47126-predictor-75575b7677-6f6r4	Scheduled	Successfully assigned e2e-gw0-a3ce8b9d69/error-404-isvc-47126-predictor-75575b7677-6f6r4 to ip-10-0-137-4.us-east-2.compute.internal
22:05:22	e2e-gw0-a3ce8b9d69		model-chainer-69f597b54c-8hb4j	Scheduled	Successfully assigned e2e-gw0-a3ce8b9d69/model-chainer-69f597b54c-8hb4j to ip-10-0-140-139.us-east-2.compute.internal
22:16:40	e2e-gw0-a3ce8b9d69		error-404-isvc-32399-predictor-96655cc85-knlc8	Scheduled	Successfully assigned e2e-gw0-a3ce8b9d69/error-404-isvc-32399-predictor-96655cc85-knlc8 to ip-10-0-137-4.us-east-2.compute.internal
22:05:41	e2e-gw0-a3ce8b9d69		error-404-isvc-e9913-predictor-6c79bcdf7f-ls4xm	Scheduled	Successfully assigned e2e-gw0-a3ce8b9d69/error-404-isvc-e9913-predictor-6c79bcdf7f-ls4xm to ip-10-0-137-4.us-east-2.compute.internal
22:03:02	e2e-gw0-a3ce8b9d69		isvc-sklearn-graph-1-predictor-5fb958cf9f-v8qlj	Scheduled	Successfully assigned e2e-gw0-a3ce8b9d69/isvc-sklearn-graph-1-predictor-5fb958cf9f-v8qlj to ip-10-0-137-4.us-east-2.compute.internal
22:03:02	e2e-gw0-a3ce8b9d69		isvc-sklearn-graph-2-predictor-5b77b5f6d8-2hwjn	Scheduled	Successfully assigned e2e-gw0-a3ce8b9d69/isvc-sklearn-graph-2-predictor-5b77b5f6d8-2hwjn to ip-10-0-132-252.us-east-2.compute.internal
22:16:23	e2e-gw0-a3ce8b9d69		sequence-graph-47126-f6975c7d9-znqqj	Scheduled	Successfully assigned e2e-gw0-a3ce8b9d69/sequence-graph-47126-f6975c7d9-znqqj to ip-10-0-132-252.us-east-2.compute.internal
22:27:21	e2e-gw0-a3ce8b9d69		switch-graph-18df3-6c66d86fbb-4qm9h	Scheduled	Successfully assigned e2e-gw0-a3ce8b9d69/switch-graph-18df3-6c66d86fbb-4qm9h to ip-10-0-132-252.us-east-2.compute.internal
22:17:50	e2e-gw0-a3ce8b9d69		sequence-graph-32399-b49696fbd-gmcwb	Scheduled	Successfully assigned e2e-gw0-a3ce8b9d69/sequence-graph-32399-b49696fbd-gmcwb to ip-10-0-132-252.us-east-2.compute.internal
22:06:51	e2e-gw0-a3ce8b9d69		sequence-graph-e9913-767b8c8c7d-kzdqj	Scheduled	Successfully assigned e2e-gw0-a3ce8b9d69/sequence-graph-e9913-767b8c8c7d-kzdqj to ip-10-0-140-139.us-east-2.compute.internal
22:26:11	e2e-gw0-a3ce8b9d69		success-200-isvc-18df3-predictor-5d95b78944-cmx2g	Scheduled	Successfully assigned e2e-gw0-a3ce8b9d69/success-200-isvc-18df3-predictor-5d95b78944-cmx2g to ip-10-0-140-139.us-east-2.compute.internal
22:16:40	e2e-gw0-a3ce8b9d69		success-200-isvc-32399-predictor-8666d8d676-jwpv4	Scheduled	Successfully assigned e2e-gw0-a3ce8b9d69/success-200-isvc-32399-predictor-8666d8d676-jwpv4 to ip-10-0-140-139.us-east-2.compute.internal
22:05:41	e2e-gw0-a3ce8b9d69		success-200-isvc-e9913-predictor-97cc787fd-c8vb2	Scheduled	Successfully assigned e2e-gw0-a3ce8b9d69/success-200-isvc-e9913-predictor-97cc787fd-c8vb2 to ip-10-0-140-139.us-east-2.compute.internal
22:03:02	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-1-predictor-5fb958cf9f-v8qlj	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:7391cf9cf76624c7a5b9b0d93ea4fff7540958210c8e04a4ba8305e117812c42" already present on machine and can be accessed by the pod
22:03:02	e2e-gw0-a3ce8b9d69	deployment-controller	isvc-sklearn-graph-1-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-graph-1-predictor-5fb958cf9f from 0 to 1
22:03:02	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-1-predictor-5fb958cf9f-v8qlj	Started	Container started
22:03:02	e2e-gw0-a3ce8b9d69	replicaset-controller	isvc-xgboost-graph-predictor-74db998dbd	SuccessfulCreate	Created pod: isvc-xgboost-graph-predictor-74db998dbd-gv7p2
22:03:02	e2e-gw0-a3ce8b9d69	deployment-controller	isvc-sklearn-graph-2-predictor	ScalingReplicaSet	Scaled up replica set isvc-sklearn-graph-2-predictor-5b77b5f6d8 from 0 to 1
22:03:02	e2e-gw0-a3ce8b9d69	deployment-controller	isvc-xgboost-graph-predictor	ScalingReplicaSet	Scaled up replica set isvc-xgboost-graph-predictor-74db998dbd from 0 to 1
22:03:02	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-1-predictor-5fb958cf9f-v8qlj	Created	Container created
22:03:02	e2e-gw0-a3ce8b9d69	replicaset-controller	isvc-sklearn-graph-2-predictor-5b77b5f6d8	SuccessfulCreate	Created pod: isvc-sklearn-graph-2-predictor-5b77b5f6d8-2hwjn
22:03:02	e2e-gw0-a3ce8b9d69	v1beta1Controllers	isvc-sklearn-graph-1	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-1": the object has been modified; please apply your changes to the latest version and try again
22:03:02	e2e-gw0-a3ce8b9d69	v1beta1Controllers	isvc-sklearn-graph-1	UpdateFailed	Failed to update status for InferenceService "isvc-sklearn-graph-1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-1": the object has been modified; please apply your changes to the latest version and try again
22:03:02	e2e-gw0-a3ce8b9d69	multus	isvc-sklearn-graph-1-predictor-5fb958cf9f-v8qlj	AddedInterface	Add eth0 [10.133.0.23/23] from ovn-kubernetes
22:03:02	e2e-gw0-a3ce8b9d69	replicaset-controller	isvc-sklearn-graph-1-predictor-5fb958cf9f	SuccessfulCreate	Created pod: isvc-sklearn-graph-1-predictor-5fb958cf9f-v8qlj
22:03:03	e2e-gw0-a3ce8b9d69	kubelet	isvc-xgboost-graph-predictor-74db998dbd-gv7p2	Pulled	Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:7391cf9cf76624c7a5b9b0d93ea4fff7540958210c8e04a4ba8305e117812c42" already present on machine and can be accessed by the pod
22:03:03	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-2-predictor-5b77b5f6d8-2hwjn	Pulling	Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:7391cf9cf76624c7a5b9b0d93ea4fff7540958210c8e04a4ba8305e117812c42"
22:03:03	e2e-gw0-a3ce8b9d69	multus	isvc-xgboost-graph-predictor-74db998dbd-gv7p2	AddedInterface	Add eth0 [10.133.0.24/23] from ovn-kubernetes
22:03:03	e2e-gw0-a3ce8b9d69	multus	isvc-sklearn-graph-2-predictor-5b77b5f6d8-2hwjn	AddedInterface	Add eth0 [10.134.0.21/23] from ovn-kubernetes
22:03:03	e2e-gw0-a3ce8b9d69	kubelet	isvc-xgboost-graph-predictor-74db998dbd-gv7p2	Created	Container created
22:03:03	e2e-gw0-a3ce8b9d69	kubelet	isvc-xgboost-graph-predictor-74db998dbd-gv7p2	Started	Container started
22:03:06	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-2-predictor-5b77b5f6d8-2hwjn	Started	Container started
22:03:06	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-2-predictor-5b77b5f6d8-2hwjn	Pulled	Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:7391cf9cf76624c7a5b9b0d93ea4fff7540958210c8e04a4ba8305e117812c42" in 3.392s (3.392s including waiting). Image size: 332165546 bytes.
22:03:06	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-2-predictor-5b77b5f6d8-2hwjn	Created	Container created
22:03:07	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-1-predictor-5fb958cf9f-v8qlj	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-2144"
22:03:08	e2e-gw0-a3ce8b9d69	kubelet	isvc-xgboost-graph-predictor-74db998dbd-gv7p2	Pulling	Pulling image "kserve/xgbserver:latest"
22:03:10	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-2-predictor-5b77b5f6d8-2hwjn	Pulling	Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-2144"
22:03:24	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-2-predictor-5b77b5f6d8-2hwjn	Started	Container started
22:03:24	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-2-predictor-5b77b5f6d8-2hwjn	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-2144" in 13.823s (13.823s including waiting). Image size: 1630130065 bytes.
22:03:24	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-2-predictor-5b77b5f6d8-2hwjn	Pulling	Pulling image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f"
22:03:24	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-2-predictor-5b77b5f6d8-2hwjn	Created	Container created
22:03:26	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-1-predictor-5fb958cf9f-v8qlj	Created	Container created
22:03:26	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-1-predictor-5fb958cf9f-v8qlj	Pulled	Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-2144" in 19.661s (19.661s including waiting). Image size: 1630130065 bytes.
22:03:26	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-2-predictor-5b77b5f6d8-2hwjn	Started	Container started
22:03:26	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-2-predictor-5b77b5f6d8-2hwjn	Created	Container created
22:03:26	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-2-predictor-5b77b5f6d8-2hwjn	Pulled	Successfully pulled image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" in 2.335s (2.335s including waiting). Image size: 184747123 bytes.
22:03:26	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-1-predictor-5fb958cf9f-v8qlj	Started	Container started
22:03:26	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-1-predictor-5fb958cf9f-v8qlj	Pulled	Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod
22:03:26	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-1-predictor-5fb958cf9f-v8qlj	Started	Container started
22:03:26	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-1-predictor-5fb958cf9f-v8qlj	Created	Container created
22:03:33	e2e-gw0-a3ce8b9d69	kubelet	isvc-xgboost-graph-predictor-74db998dbd-gv7p2	Created	Container created
22:03:33	e2e-gw0-a3ce8b9d69	kubelet	isvc-xgboost-graph-predictor-74db998dbd-gv7p2	Pulled	Successfully pulled image "kserve/xgbserver:latest" in 24.952s (24.952s including waiting). Image size: 1460330245 bytes.
22:03:33	e2e-gw0-a3ce8b9d69	kubelet	isvc-xgboost-graph-predictor-74db998dbd-gv7p2	Started	Container started
22:03:33	e2e-gw0-a3ce8b9d69	kubelet	isvc-xgboost-graph-predictor-74db998dbd-gv7p2	Pulled	Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod
22:03:33	e2e-gw0-a3ce8b9d69	kubelet	isvc-xgboost-graph-predictor-74db998dbd-gv7p2	Created	Container created
22:03:33	e2e-gw0-a3ce8b9d69	kubelet	isvc-xgboost-graph-predictor-74db998dbd-gv7p2	Started	Container started
22:03:47 (x3)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:03:47 (x3)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:04:02 (x4)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:04:02 (x3)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:04:02 (x3)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:04:02 (x4)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:04:40 (x9)	e2e-gw0-a3ce8b9d69	kubelet	isvc-xgboost-graph-predictor-74db998dbd-gv7p2	Unhealthy	Readiness probe failed: dial tcp 10.133.0.24:8080: connect: connection refused
22:04:44 (x9)	e2e-gw0-a3ce8b9d69	v1beta1Controllers	isvc-sklearn-graph-2	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:04:44	e2e-gw0-a3ce8b9d69	v1beta1Controllers	isvc-sklearn-graph-2	InferenceServiceReady	InferenceService [isvc-sklearn-graph-2] is Ready
22:04:50	e2e-gw0-a3ce8b9d69	v1beta1Controllers	isvc-xgboost-graph	InferenceServiceReady	InferenceService [isvc-xgboost-graph] is Ready
22:04:50 (x9)	e2e-gw0-a3ce8b9d69	v1beta1Controllers	isvc-xgboost-graph	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:04:56	e2e-gw0-a3ce8b9d69	v1beta1Controllers	isvc-sklearn-graph-1	InferenceServiceReady	InferenceService [isvc-sklearn-graph-1] is Ready
22:04:56 (x9)	e2e-gw0-a3ce8b9d69	v1beta1Controllers	isvc-sklearn-graph-1	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:05:02 (x4)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:05:02 (x4)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	isvc-sklearn-graph-2-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:05:17 (x6)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:05:17 (x6)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:05:17 (x6)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	isvc-sklearn-graph-1-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:05:17 (x6)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	isvc-xgboost-graph-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:05:22	e2e-gw0-a3ce8b9d69	deployment-controller	model-chainer	ScalingReplicaSet	Scaled up replica set model-chainer-69f597b54c from 0 to 1
22:05:22	e2e-gw0-a3ce8b9d69	kubelet	model-chainer-69f597b54c-8hb4j	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:7e2799a8e0da0716aa409dee4f019d9799355342f3f8e0102d89de7049943ac0" already present on machine and can be accessed by the pod
22:05:22	e2e-gw0-a3ce8b9d69	multus	model-chainer-69f597b54c-8hb4j	AddedInterface	Add eth0 [10.132.0.36/23] from ovn-kubernetes
22:05:22	e2e-gw0-a3ce8b9d69	kubelet	model-chainer-69f597b54c-8hb4j	Created	Container created
22:05:22	e2e-gw0-a3ce8b9d69	kubelet	model-chainer-69f597b54c-8hb4j	Started	Container started
22:05:22	e2e-gw0-a3ce8b9d69	replicaset-controller	model-chainer-69f597b54c	SuccessfulCreate	Created pod: model-chainer-69f597b54c-8hb4j
22:05:22	e2e-gw0-a3ce8b9d69	InferenceGraphController	model-chainer	InferenceGraphReady	InferenceGraph [model-chainer] is Ready
22:05:34	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-1-predictor-5fb958cf9f-v8qlj	Killing	Stopping container kube-rbac-proxy
22:05:34	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-2-predictor-5b77b5f6d8-2hwjn	Killing	Stopping container kube-rbac-proxy
22:05:34	e2e-gw0-a3ce8b9d69	kubelet	isvc-xgboost-graph-predictor-74db998dbd-gv7p2	Killing	Stopping container kube-rbac-proxy
22:05:34	e2e-gw0-a3ce8b9d69	kubelet	isvc-xgboost-graph-predictor-74db998dbd-gv7p2	Killing	Stopping container kserve-container
22:05:34 (x10)	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-2-predictor-5b77b5f6d8-2hwjn	Unhealthy	Readiness probe failed: dial tcp 10.134.0.21:8080: connect: connection refused
22:05:34	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-2-predictor-5b77b5f6d8-2hwjn	Unhealthy	Readiness probe failed: Get "https://10.134.0.21:8643/healthz": dial tcp 10.134.0.21:8643: connect: connection refused
22:05:34	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-2-predictor-5b77b5f6d8-2hwjn	Killing	Stopping container kserve-container
22:05:34	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-1-predictor-5fb958cf9f-v8qlj	Killing	Stopping container kserve-container
22:05:35 (x13)	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-1-predictor-5fb958cf9f-v8qlj	Unhealthy	Readiness probe failed: dial tcp 10.133.0.23:8080: connect: connection refused
22:05:35	e2e-gw0-a3ce8b9d69	kubelet	isvc-xgboost-graph-predictor-74db998dbd-gv7p2	Unhealthy	Readiness probe failed: Get "https://10.133.0.24:8643/healthz": dial tcp 10.133.0.24:8643: connect: connection refused
22:05:39 (x2)	e2e-gw0-a3ce8b9d69	kubelet	isvc-sklearn-graph-1-predictor-5fb958cf9f-v8qlj	Unhealthy	Readiness probe failed: Get "https://10.133.0.23:8643/healthz": dial tcp 10.133.0.23:8643: connect: connection refused
22:05:41	e2e-gw0-a3ce8b9d69	v1beta1Controllers	success-200-isvc-e9913	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-e9913": the object has been modified; please apply your changes to the latest version and try again
22:05:41	e2e-gw0-a3ce8b9d69	replicaset-controller	error-404-isvc-e9913-predictor-6c79bcdf7f	SuccessfulCreate	Created pod: error-404-isvc-e9913-predictor-6c79bcdf7f-ls4xm
22:05:41	e2e-gw0-a3ce8b9d69	v1beta1Controllers	success-200-isvc-e9913	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-e9913": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-e9913": the object has been modified; please apply your changes to the latest version and try again
22:05:41	e2e-gw0-a3ce8b9d69	deployment-controller	success-200-isvc-e9913-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-e9913-predictor-97cc787fd from 0 to 1
22:05:41	e2e-gw0-a3ce8b9d69	replicaset-controller	success-200-isvc-e9913-predictor-97cc787fd	SuccessfulCreate	Created pod: success-200-isvc-e9913-predictor-97cc787fd-c8vb2
22:05:41	e2e-gw0-a3ce8b9d69	deployment-controller	error-404-isvc-e9913-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-e9913-predictor-6c79bcdf7f from 0 to 1
22:05:42	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-e9913-predictor-6c79bcdf7f-ls4xm	Started	Container started
22:05:42	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-e9913-predictor-97cc787fd-c8vb2	Pulled	Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod
22:05:42	e2e-gw0-a3ce8b9d69	multus	error-404-isvc-e9913-predictor-6c79bcdf7f-ls4xm	AddedInterface	Add eth0 [10.133.0.27/23] from ovn-kubernetes
22:05:42	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-e9913-predictor-6c79bcdf7f-ls4xm	Created	Container created
22:05:42	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-e9913-predictor-6c79bcdf7f-ls4xm	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-2144" already present on machine and can be accessed by the pod
22:05:42	e2e-gw0-a3ce8b9d69	multus	success-200-isvc-e9913-predictor-97cc787fd-c8vb2	AddedInterface	Add eth0 [10.132.0.37/23] from ovn-kubernetes
22:05:42	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-e9913-predictor-97cc787fd-c8vb2	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-2144" already present on machine and can be accessed by the pod
22:05:42	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-e9913-predictor-97cc787fd-c8vb2	Created	Container created
22:05:42	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-e9913-predictor-97cc787fd-c8vb2	Started	Container started
22:05:42	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-e9913-predictor-6c79bcdf7f-ls4xm	Pulled	Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod
22:05:42	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-e9913-predictor-97cc787fd-c8vb2	Created	Container created
22:05:42	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-e9913-predictor-97cc787fd-c8vb2	Started	Container started
22:05:42	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-e9913-predictor-6c79bcdf7f-ls4xm	Started	Container started
22:05:42	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-e9913-predictor-6c79bcdf7f-ls4xm	Created	Container created
22:05:52 (x2)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	model-chainer	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:05:52 (x2)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	model-chainer	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:05:56	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	success-200-isvc-e9913-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:05:56	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	success-200-isvc-e9913-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:06:26 (x3)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	error-404-isvc-e9913-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:06:26 (x3)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	error-404-isvc-e9913-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:06:38	e2e-gw0-a3ce8b9d69	v1beta1Controllers	success-200-isvc-e9913	InferenceServiceReady	InferenceService [success-200-isvc-e9913] is Ready
22:06:39 (x5)	e2e-gw0-a3ce8b9d69	v1beta1Controllers	error-404-isvc-e9913	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:06:39 (x5)	e2e-gw0-a3ce8b9d69	v1beta1Controllers	success-200-isvc-e9913	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:06:39	e2e-gw0-a3ce8b9d69	v1beta1Controllers	error-404-isvc-e9913	InferenceServiceReady	InferenceService [error-404-isvc-e9913] is Ready
22:06:51	e2e-gw0-a3ce8b9d69	replicaset-controller	sequence-graph-e9913-767b8c8c7d	SuccessfulCreate	Created pod: sequence-graph-e9913-767b8c8c7d-kzdqj
22:06:51	e2e-gw0-a3ce8b9d69	InferenceGraphController	sequence-graph-e9913	InferenceGraphReady	InferenceGraph [sequence-graph-e9913] is Ready
22:06:51	e2e-gw0-a3ce8b9d69	deployment-controller	sequence-graph-e9913	ScalingReplicaSet	Scaled up replica set sequence-graph-e9913-767b8c8c7d from 0 to 1
22:06:52	e2e-gw0-a3ce8b9d69	multus	sequence-graph-e9913-767b8c8c7d-kzdqj	AddedInterface	Add eth0 [10.132.0.39/23] from ovn-kubernetes
22:06:52	e2e-gw0-a3ce8b9d69	kubelet	sequence-graph-e9913-767b8c8c7d-kzdqj	Started	Container started
22:06:52	e2e-gw0-a3ce8b9d69	kubelet	sequence-graph-e9913-767b8c8c7d-kzdqj	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:7e2799a8e0da0716aa409dee4f019d9799355342f3f8e0102d89de7049943ac0" already present on machine and can be accessed by the pod
22:06:52	e2e-gw0-a3ce8b9d69	kubelet	sequence-graph-e9913-767b8c8c7d-kzdqj	Created	Container created
22:07:06	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	sequence-graph-e9913	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:07:06	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	sequence-graph-e9913	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:07:11 (x5)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	success-200-isvc-e9913-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:07:11 (x3)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	error-404-isvc-e9913-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:07:11 (x3)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	error-404-isvc-e9913-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:07:11 (x5)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	success-200-isvc-e9913-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:15:07	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-e9913-predictor-6c79bcdf7f-ls4xm	Killing	Stopping container kserve-container
22:15:07	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-e9913-predictor-6c79bcdf7f-ls4xm	Killing	Stopping container kube-rbac-proxy
22:15:07	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-e9913-predictor-97cc787fd-c8vb2	Killing	Stopping container kserve-container
22:15:07	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-e9913-predictor-97cc787fd-c8vb2	Killing	Stopping container kube-rbac-proxy
22:15:08 (x8)	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-e9913-predictor-97cc787fd-c8vb2	Unhealthy	Readiness probe failed: dial tcp 10.132.0.37:8080: connect: connection refused
22:15:08	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-e9913-predictor-97cc787fd-c8vb2	Unhealthy	Readiness probe failed: Get "https://10.132.0.37:8643/healthz": dial tcp 10.132.0.37:8643: connect: connection refused
22:15:09	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-e9913-predictor-6c79bcdf7f-ls4xm	Unhealthy	Readiness probe failed: Get "https://10.133.0.27:8643/healthz": dial tcp 10.133.0.27:8643: connect: connection refused
22:15:09 (x8)	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-e9913-predictor-6c79bcdf7f-ls4xm	Unhealthy	Readiness probe failed: dial tcp 10.133.0.27:8080: connect: connection refused
22:15:13	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-47126-predictor-77fcdcc8cb-zwn9w	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-2144" already present on machine and can be accessed by the pod
22:15:13	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-47126-predictor-77fcdcc8cb-zwn9w	Started	Container started
22:15:13	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-47126-predictor-75575b7677-6f6r4	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-2144" already present on machine and can be accessed by the pod
22:15:13	e2e-gw0-a3ce8b9d69	deployment-controller	error-404-isvc-47126-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-47126-predictor-75575b7677 from 0 to 1
22:15:13	e2e-gw0-a3ce8b9d69	replicaset-controller	error-404-isvc-47126-predictor-75575b7677	SuccessfulCreate	Created pod: error-404-isvc-47126-predictor-75575b7677-6f6r4
22:15:13	e2e-gw0-a3ce8b9d69	multus	success-200-isvc-47126-predictor-77fcdcc8cb-zwn9w	AddedInterface	Add eth0 [10.132.0.42/23] from ovn-kubernetes
22:15:13	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-47126-predictor-77fcdcc8cb-zwn9w	Created	Container created
22:15:13	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-47126-predictor-77fcdcc8cb-zwn9w	Started	Container started
22:15:13	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-47126-predictor-77fcdcc8cb-zwn9w	Pulled	Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod
22:15:13	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-47126-predictor-77fcdcc8cb-zwn9w	Created	Container created
22:15:13	e2e-gw0-a3ce8b9d69	multus	error-404-isvc-47126-predictor-75575b7677-6f6r4	AddedInterface	Add eth0 [10.133.0.29/23] from ovn-kubernetes
22:15:13	e2e-gw0-a3ce8b9d69	v1beta1Controllers	success-200-isvc-47126	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-47126": the object has been modified; please apply your changes to the latest version and try again
22:15:13	e2e-gw0-a3ce8b9d69	v1beta1Controllers	success-200-isvc-47126	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-47126": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-47126": the object has been modified; please apply your changes to the latest version and try again
22:15:13	e2e-gw0-a3ce8b9d69	replicaset-controller	success-200-isvc-47126-predictor-77fcdcc8cb	SuccessfulCreate	Created pod: success-200-isvc-47126-predictor-77fcdcc8cb-zwn9w
22:15:13	e2e-gw0-a3ce8b9d69	deployment-controller	success-200-isvc-47126-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-47126-predictor-77fcdcc8cb from 0 to 1
22:15:14	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-47126-predictor-75575b7677-6f6r4	Started	Container started
22:15:14	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-47126-predictor-75575b7677-6f6r4	Created	Container created
22:15:14	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-47126-predictor-75575b7677-6f6r4	Pulled	Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod
22:15:14	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-47126-predictor-75575b7677-6f6r4	Started	Container started
22:15:14	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-47126-predictor-75575b7677-6f6r4	Created	Container created
22:15:43	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	error-404-isvc-47126-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:15:43	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	error-404-isvc-47126-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:16:00 (x7)	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-47126-predictor-75575b7677-6f6r4	Unhealthy	Readiness probe failed: dial tcp 10.133.0.29:8080: connect: connection refused
22:16:01 (x7)	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-47126-predictor-77fcdcc8cb-zwn9w	Unhealthy	Readiness probe failed: dial tcp 10.132.0.42:8080: connect: connection refused
22:16:11	e2e-gw0-a3ce8b9d69	v1beta1Controllers	error-404-isvc-47126	InferenceServiceReady	InferenceService [error-404-isvc-47126] is Ready
22:16:11 (x5)	e2e-gw0-a3ce8b9d69	v1beta1Controllers	error-404-isvc-47126	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:16:11 (x5)	e2e-gw0-a3ce8b9d69	v1beta1Controllers	success-200-isvc-47126	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:16:11	e2e-gw0-a3ce8b9d69	v1beta1Controllers	success-200-isvc-47126	InferenceServiceReady	InferenceService [success-200-isvc-47126] is Ready
22:16:23	e2e-gw0-a3ce8b9d69	kubelet	sequence-graph-47126-f6975c7d9-znqqj	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:7e2799a8e0da0716aa409dee4f019d9799355342f3f8e0102d89de7049943ac0" already present on machine and can be accessed by the pod
22:16:23	e2e-gw0-a3ce8b9d69	multus	sequence-graph-47126-f6975c7d9-znqqj	AddedInterface	Add eth0 [10.134.0.23/23] from ovn-kubernetes
22:16:23	e2e-gw0-a3ce8b9d69	kubelet	sequence-graph-47126-f6975c7d9-znqqj	Created	Container created
22:16:23	e2e-gw0-a3ce8b9d69	kubelet	sequence-graph-47126-f6975c7d9-znqqj	Started	Container started
22:16:23	e2e-gw0-a3ce8b9d69	replicaset-controller	sequence-graph-47126-f6975c7d9	SuccessfulCreate	Created pod: sequence-graph-47126-f6975c7d9-znqqj
22:16:23	e2e-gw0-a3ce8b9d69	deployment-controller	sequence-graph-47126	ScalingReplicaSet	Scaled up replica set sequence-graph-47126-f6975c7d9 from 0 to 1
22:16:23	e2e-gw0-a3ce8b9d69	InferenceGraphController	sequence-graph-47126	InferenceGraphReady	InferenceGraph [sequence-graph-47126] is Ready
22:16:28 (x4)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	error-404-isvc-47126-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:16:28 (x5)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	success-200-isvc-47126-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:16:28 (x5)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	success-200-isvc-47126-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:16:28 (x4)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	error-404-isvc-47126-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:16:34	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-47126-predictor-77fcdcc8cb-zwn9w	Killing	Stopping container kserve-container
22:16:34	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-47126-predictor-75575b7677-6f6r4	Killing	Stopping container kserve-container
22:16:34	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-47126-predictor-77fcdcc8cb-zwn9w	Killing	Stopping container kube-rbac-proxy
22:16:34	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-47126-predictor-75575b7677-6f6r4	Killing	Stopping container kube-rbac-proxy
22:16:35	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-47126-predictor-75575b7677-6f6r4	Unhealthy	Readiness probe failed: Get "https://10.133.0.29:8643/healthz": dial tcp 10.133.0.29:8643: connect: connection refused
22:16:36	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-47126-predictor-77fcdcc8cb-zwn9w	Unhealthy	Readiness probe failed: Get "https://10.132.0.42:8643/healthz": dial tcp 10.132.0.42:8643: connect: connection refused
22:16:39	e2e-gw0-a3ce8b9d69	deployment-controller	success-200-isvc-32399-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-32399-predictor-8666d8d676 from 0 to 1
22:16:39	e2e-gw0-a3ce8b9d69	v1beta1Controllers	success-200-isvc-32399	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-32399": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-32399": the object has been modified; please apply your changes to the latest version and try again
22:16:39	e2e-gw0-a3ce8b9d69	v1beta1Controllers	success-200-isvc-32399	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-32399": the object has been modified; please apply your changes to the latest version and try again
22:16:40	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-32399-predictor-96655cc85-knlc8	Started	Container started
22:16:40	e2e-gw0-a3ce8b9d69	deployment-controller	error-404-isvc-32399-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-32399-predictor-96655cc85 from 0 to 1
22:16:40	e2e-gw0-a3ce8b9d69	replicaset-controller	error-404-isvc-32399-predictor-96655cc85	SuccessfulCreate	Created pod: error-404-isvc-32399-predictor-96655cc85-knlc8
22:16:40	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-32399-predictor-96655cc85-knlc8	Started	Container started
22:16:40	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-32399-predictor-96655cc85-knlc8	Created	Container created
22:16:40	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-32399-predictor-96655cc85-knlc8	Pulled	Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod
22:16:40	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-32399-predictor-96655cc85-knlc8	Created	Container created
22:16:40	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-32399-predictor-96655cc85-knlc8	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-2144" already present on machine and can be accessed by the pod
22:16:40	e2e-gw0-a3ce8b9d69	multus	error-404-isvc-32399-predictor-96655cc85-knlc8	AddedInterface	Add eth0 [10.133.0.31/23] from ovn-kubernetes
22:16:40	e2e-gw0-a3ce8b9d69	replicaset-controller	success-200-isvc-32399-predictor-8666d8d676	SuccessfulCreate	Created pod: success-200-isvc-32399-predictor-8666d8d676-jwpv4
22:16:40	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-32399-predictor-8666d8d676-jwpv4	Started	Container started
22:16:40	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-32399-predictor-8666d8d676-jwpv4	Created	Container created
22:16:40	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-32399-predictor-8666d8d676-jwpv4	Pulled	Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod
22:16:40	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-32399-predictor-8666d8d676-jwpv4	Started	Container started
22:16:40	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-32399-predictor-8666d8d676-jwpv4	Created	Container created
22:16:40	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-32399-predictor-8666d8d676-jwpv4	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-2144" already present on machine and can be accessed by the pod
22:16:40	e2e-gw0-a3ce8b9d69	multus	success-200-isvc-32399-predictor-8666d8d676-jwpv4	AddedInterface	Add eth0 [10.132.0.44/23] from ovn-kubernetes
22:16:54	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	success-200-isvc-32399-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:16:54	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	success-200-isvc-32399-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:16:55	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	error-404-isvc-32399-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:16:55	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	error-404-isvc-32399-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:17:23 (x4)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	sequence-graph-47126	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:17:23 (x4)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	sequence-graph-47126	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:17:37	e2e-gw0-a3ce8b9d69	v1beta1Controllers	success-200-isvc-32399	InferenceServiceReady	InferenceService [success-200-isvc-32399] is Ready
22:17:37 (x5)	e2e-gw0-a3ce8b9d69	v1beta1Controllers	success-200-isvc-32399	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:17:38 (x5)	e2e-gw0-a3ce8b9d69	v1beta1Controllers	error-404-isvc-32399	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:17:38	e2e-gw0-a3ce8b9d69	v1beta1Controllers	error-404-isvc-32399	InferenceServiceReady	InferenceService [error-404-isvc-32399] is Ready
22:17:50	e2e-gw0-a3ce8b9d69	replicaset-controller	sequence-graph-32399-b49696fbd	SuccessfulCreate	Created pod: sequence-graph-32399-b49696fbd-gmcwb
22:17:50	e2e-gw0-a3ce8b9d69	kubelet	sequence-graph-32399-b49696fbd-gmcwb	Created	Container created
22:17:50	e2e-gw0-a3ce8b9d69	deployment-controller	sequence-graph-32399	ScalingReplicaSet	Scaled up replica set sequence-graph-32399-b49696fbd from 0 to 1
22:17:50	e2e-gw0-a3ce8b9d69	InferenceGraphController	sequence-graph-32399	InferenceGraphReady	InferenceGraph [sequence-graph-32399] is Ready
22:17:50	e2e-gw0-a3ce8b9d69	multus	sequence-graph-32399-b49696fbd-gmcwb	AddedInterface	Add eth0 [10.134.0.25/23] from ovn-kubernetes
22:17:50	e2e-gw0-a3ce8b9d69	kubelet	sequence-graph-32399-b49696fbd-gmcwb	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:7e2799a8e0da0716aa409dee4f019d9799355342f3f8e0102d89de7049943ac0" already present on machine and can be accessed by the pod
22:17:50	e2e-gw0-a3ce8b9d69	kubelet	sequence-graph-32399-b49696fbd-gmcwb	Started	Container started
22:18:10 (x5)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	success-200-isvc-32399-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:18:10 (x5)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	error-404-isvc-32399-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:18:10 (x5)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	success-200-isvc-32399-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:18:10 (x5)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	error-404-isvc-32399-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:18:20 (x2)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	sequence-graph-32399	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:18:20 (x2)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	sequence-graph-32399	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:26:05	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-32399-predictor-8666d8d676-jwpv4	Killing	Stopping container kserve-container
22:26:05	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-32399-predictor-8666d8d676-jwpv4	Killing	Stopping container kube-rbac-proxy
22:26:05	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-32399-predictor-96655cc85-knlc8	Killing	Stopping container kserve-container
22:26:05	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-32399-predictor-96655cc85-knlc8	Killing	Stopping container kube-rbac-proxy
22:26:07	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-32399-predictor-8666d8d676-jwpv4	Unhealthy	Readiness probe failed: Get "https://10.132.0.44:8643/healthz": dial tcp 10.132.0.44:8643: connect: connection refused
22:26:07 (x8)	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-32399-predictor-8666d8d676-jwpv4	Unhealthy	Readiness probe failed: dial tcp 10.132.0.44:8080: connect: connection refused
22:26:08 (x8)	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-32399-predictor-96655cc85-knlc8	Unhealthy	Readiness probe failed: dial tcp 10.133.0.31:8080: connect: connection refused
22:26:08	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-32399-predictor-96655cc85-knlc8	Unhealthy	Readiness probe failed: Get "https://10.133.0.31:8643/healthz": dial tcp 10.133.0.31:8643: connect: connection refused
22:26:10	e2e-gw0-a3ce8b9d69	v1beta1Controllers	success-200-isvc-18df3	UpdateFailed	Failed to update status for InferenceService "success-200-isvc-18df3": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-18df3": the object has been modified; please apply your changes to the latest version and try again
22:26:10	e2e-gw0-a3ce8b9d69	v1beta1Controllers	success-200-isvc-18df3	InternalError	fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-18df3": the object has been modified; please apply your changes to the latest version and try again
22:26:10	e2e-gw0-a3ce8b9d69	deployment-controller	success-200-isvc-18df3-predictor	ScalingReplicaSet	Scaled up replica set success-200-isvc-18df3-predictor-5d95b78944 from 0 to 1
22:26:11	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-18df3-predictor-5d95b78944-cmx2g	Created	Container created
22:26:11	e2e-gw0-a3ce8b9d69	replicaset-controller	error-404-isvc-18df3-predictor-7b5568b5d9	SuccessfulCreate	Created pod: error-404-isvc-18df3-predictor-7b5568b5d9-k48m5
22:26:11	e2e-gw0-a3ce8b9d69	multus	error-404-isvc-18df3-predictor-7b5568b5d9-k48m5	AddedInterface	Add eth0 [10.133.0.33/23] from ovn-kubernetes
22:26:11	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-18df3-predictor-7b5568b5d9-k48m5	Pulled	Container image "quay.io/opendatahub/error-404-isvc:odh-pr-2144" already present on machine and can be accessed by the pod
22:26:11	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-18df3-predictor-5d95b78944-cmx2g	Created	Container created
22:26:11	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-18df3-predictor-5d95b78944-cmx2g	Started	Container started
22:26:11	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-18df3-predictor-7b5568b5d9-k48m5	Created	Container created
22:26:11	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-18df3-predictor-5d95b78944-cmx2g	Started	Container started
22:26:11	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-18df3-predictor-7b5568b5d9-k48m5	Started	Container started
22:26:11	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-18df3-predictor-5d95b78944-cmx2g	Pulled	Container image "quay.io/opendatahub/success-200-isvc:odh-pr-2144" already present on machine and can be accessed by the pod
22:26:11	e2e-gw0-a3ce8b9d69	multus	success-200-isvc-18df3-predictor-5d95b78944-cmx2g	AddedInterface	Add eth0 [10.132.0.46/23] from ovn-kubernetes
22:26:11	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-18df3-predictor-7b5568b5d9-k48m5	Pulled	Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod
22:26:11	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-18df3-predictor-7b5568b5d9-k48m5	Created	Container created
22:26:11	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-18df3-predictor-5d95b78944-cmx2g	Pulled	Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod
22:26:11	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-18df3-predictor-7b5568b5d9-k48m5	Started	Container started
22:26:11	e2e-gw0-a3ce8b9d69	replicaset-controller	success-200-isvc-18df3-predictor-5d95b78944	SuccessfulCreate	Created pod: success-200-isvc-18df3-predictor-5d95b78944-cmx2g
22:26:11	e2e-gw0-a3ce8b9d69	deployment-controller	error-404-isvc-18df3-predictor	ScalingReplicaSet	Scaled up replica set error-404-isvc-18df3-predictor-7b5568b5d9 from 0 to 1
22:26:25	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	success-200-isvc-18df3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:26:25	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	success-200-isvc-18df3-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:26:56 (x3)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	error-404-isvc-18df3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:26:56 (x3)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	error-404-isvc-18df3-predictor	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:26:58 (x7)	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-18df3-predictor-7b5568b5d9-k48m5	Unhealthy	Readiness probe failed: dial tcp 10.133.0.33:8080: connect: connection refused
22:26:58 (x7)	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-18df3-predictor-5d95b78944-cmx2g	Unhealthy	Readiness probe failed: dial tcp 10.132.0.46:8080: connect: connection refused
22:27:08 (x5)	e2e-gw0-a3ce8b9d69	v1beta1Controllers	success-200-isvc-18df3	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:27:08	e2e-gw0-a3ce8b9d69	v1beta1Controllers	success-200-isvc-18df3	InferenceServiceReady	InferenceService [success-200-isvc-18df3] is Ready
22:27:08	e2e-gw0-a3ce8b9d69	v1beta1Controllers	error-404-isvc-18df3	InferenceServiceReady	InferenceService [error-404-isvc-18df3] is Ready
22:27:08 (x5)	e2e-gw0-a3ce8b9d69	v1beta1Controllers	error-404-isvc-18df3	VirtualServiceCRDNotFound	Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
22:27:21	e2e-gw0-a3ce8b9d69	deployment-controller	switch-graph-18df3	ScalingReplicaSet	Scaled up replica set switch-graph-18df3-6c66d86fbb from 0 to 1
22:27:21	e2e-gw0-a3ce8b9d69	replicaset-controller	switch-graph-18df3-6c66d86fbb	SuccessfulCreate	Created pod: switch-graph-18df3-6c66d86fbb-4qm9h
22:27:21	e2e-gw0-a3ce8b9d69	kubelet	switch-graph-18df3-6c66d86fbb-4qm9h	Started	Container started
22:27:21	e2e-gw0-a3ce8b9d69	InferenceGraphController	switch-graph-18df3	InferenceGraphReady	InferenceGraph [switch-graph-18df3] is Ready
22:27:21	e2e-gw0-a3ce8b9d69	kubelet	switch-graph-18df3-6c66d86fbb-4qm9h	Created	Container created
22:27:21	e2e-gw0-a3ce8b9d69	kubelet	switch-graph-18df3-6c66d86fbb-4qm9h	Pulled	Container image "quay.io/opendatahub/kserve-router@sha256:7e2799a8e0da0716aa409dee4f019d9799355342f3f8e0102d89de7049943ac0" already present on machine and can be accessed by the pod
22:27:21	e2e-gw0-a3ce8b9d69	multus	switch-graph-18df3-6c66d86fbb-4qm9h	AddedInterface	Add eth0 [10.134.0.26/23] from ovn-kubernetes
22:27:36	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	switch-graph-18df3	FailedGetResourceMetric	failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:27:36	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	switch-graph-18df3	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
22:27:41 (x3)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	error-404-isvc-18df3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:27:41 (x5)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	success-200-isvc-18df3-predictor	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:27:41 (x5)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	success-200-isvc-18df3-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:27:41 (x3)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	error-404-isvc-18df3-predictor	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:43:42	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-18df3-predictor-5d95b78944-cmx2g	Killing	Stopping container kube-rbac-proxy
22:43:42	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-18df3-predictor-7b5568b5d9-k48m5	Killing	Stopping container kserve-container
22:43:42	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-18df3-predictor-7b5568b5d9-k48m5	Killing	Stopping container kube-rbac-proxy
22:43:42	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-18df3-predictor-5d95b78944-cmx2g	Killing	Stopping container kserve-container
22:43:43	e2e-gw0-a3ce8b9d69	kubelet	error-404-isvc-18df3-predictor-7b5568b5d9-k48m5	Unhealthy	Readiness probe failed: Get "https://10.133.0.33:8643/healthz": dial tcp 10.133.0.33:8643: connect: connection refused
22:43:43	e2e-gw0-a3ce8b9d69	kubelet	success-200-isvc-18df3-predictor-5d95b78944-cmx2g	Unhealthy	Readiness probe failed: Get "https://10.132.0.46:8643/healthz": dial tcp 10.132.0.46:8643: connect: connection refused
22:44:08 (x3)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	sequence-graph-e9913	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:44:08 (x3)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	sequence-graph-e9913	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:44:22 (x3)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	switch-graph-18df3	FailedGetResourceMetric	failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
22:44:22 (x3)	e2e-gw0-a3ce8b9d69	horizontal-pod-autoscaler	switch-graph-18df3	FailedComputeMetricsReplicas	invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)