Time Namespace Component RelatedObject Reason Message

kserve-ci-e2e-test

model-chainer-884b78d8f-rz4zw

Scheduled

Successfully assigned kserve-ci-e2e-test/model-chainer-884b78d8f-rz4zw to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

success-200-isvc-5e299-predictor-7d5dc4956b-d27ll

Scheduled

Successfully assigned kserve-ci-e2e-test/success-200-isvc-5e299-predictor-7d5dc4956b-d27ll to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

error-404-isvc-8493d-predictor-788f857f9f-wlsz5

Scheduled

Successfully assigned kserve-ci-e2e-test/error-404-isvc-8493d-predictor-788f857f9f-wlsz5 to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

isvc-xgboost-graph-predictor-784cb989b8-4c5tl

Scheduled

Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-predictor-784cb989b8-4c5tl to ip-10-0-128-241.ec2.internal

kserve-ci-e2e-test

error-404-isvc-5e299-predictor-75c4868959-6fvvq

Scheduled

Successfully assigned kserve-ci-e2e-test/error-404-isvc-5e299-predictor-75c4868959-6fvvq to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

error-404-isvc-d60f9-predictor-bcfc89bfc-f6jf6

Scheduled

Successfully assigned kserve-ci-e2e-test/error-404-isvc-d60f9-predictor-bcfc89bfc-f6jf6 to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

sequence-graph-54da1-594f6b7df8-f9cxp

Scheduled

Successfully assigned kserve-ci-e2e-test/sequence-graph-54da1-594f6b7df8-f9cxp to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

sequence-graph-5e299-6f5777484f-5l654

Scheduled

Successfully assigned kserve-ci-e2e-test/sequence-graph-5e299-6f5777484f-5l654 to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

sequence-graph-f6449-6ff9b5b46f-jl9sk

Scheduled

Successfully assigned kserve-ci-e2e-test/sequence-graph-f6449-6ff9b5b46f-jl9sk to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

error-404-isvc-d9a0e-predictor-8b7fd8fb9-nvzd7

Scheduled

Successfully assigned kserve-ci-e2e-test/error-404-isvc-d9a0e-predictor-8b7fd8fb9-nvzd7 to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

splitter-graph-3b160-555ff4db4b-rhktz

Scheduled

Successfully assigned kserve-ci-e2e-test/splitter-graph-3b160-555ff4db4b-rhktz to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

error-404-isvc-ed576-predictor-65c7fd45bf-jfsc8

Scheduled

Successfully assigned kserve-ci-e2e-test/error-404-isvc-ed576-predictor-65c7fd45bf-jfsc8 to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

error-404-isvc-f6449-predictor-5db958dfd5-jktbg

Scheduled

Successfully assigned kserve-ci-e2e-test/error-404-isvc-f6449-predictor-5db958dfd5-jktbg to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

switch-graph-d60f9-db987dd58-m2b28

Scheduled

Successfully assigned kserve-ci-e2e-test/switch-graph-d60f9-db987dd58-m2b28 to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

switch-graph-8e251-5896fcfc4c-jr8mf

Scheduled

Successfully assigned kserve-ci-e2e-test/switch-graph-8e251-5896fcfc4c-jr8mf to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

success-200-isvc-f6449-predictor-67b49d9f68-jvnsv

Scheduled

Successfully assigned kserve-ci-e2e-test/success-200-isvc-f6449-predictor-67b49d9f68-jvnsv to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

success-200-isvc-ed576-predictor-78dc55c9fb-zr27l

Scheduled

Successfully assigned kserve-ci-e2e-test/success-200-isvc-ed576-predictor-78dc55c9fb-zr27l to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

success-200-isvc-d9a0e-predictor-8587d49fdd-wpj2j

Scheduled

Successfully assigned kserve-ci-e2e-test/success-200-isvc-d9a0e-predictor-8587d49fdd-wpj2j to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

isvc-sklearn-graph-1-predictor-5466bc54c4-p8jst

Scheduled

Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-5466bc54c4-p8jst to ip-10-0-128-241.ec2.internal

kserve-ci-e2e-test

splitter-graph-c5b1b-59bd77c885-m9xp8

Scheduled

Successfully assigned kserve-ci-e2e-test/splitter-graph-c5b1b-59bd77c885-m9xp8 to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

error-404-isvc-c5b1b-predictor-78c899d8d5-r45k2

Scheduled

Successfully assigned kserve-ci-e2e-test/error-404-isvc-c5b1b-predictor-78c899d8d5-r45k2 to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

success-200-isvc-3b160-predictor-59b6c8c9d-6ksnz

Scheduled

Successfully assigned kserve-ci-e2e-test/success-200-isvc-3b160-predictor-59b6c8c9d-6ksnz to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

error-404-isvc-3b160-predictor-788d9b5895-7hb2d

Scheduled

Successfully assigned kserve-ci-e2e-test/error-404-isvc-3b160-predictor-788d9b5895-7hb2d to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

success-200-isvc-54da1-predictor-59846b8c76-rhmxv

Scheduled

Successfully assigned kserve-ci-e2e-test/success-200-isvc-54da1-predictor-59846b8c76-rhmxv to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

success-200-isvc-8493d-predictor-77b4959486-7d762

Scheduled

Successfully assigned kserve-ci-e2e-test/success-200-isvc-8493d-predictor-77b4959486-7d762 to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

success-200-isvc-8e251-predictor-59cb876bdd-dx2p4

Scheduled

Successfully assigned kserve-ci-e2e-test/success-200-isvc-8e251-predictor-59cb876bdd-dx2p4 to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

isvc-sklearn-graph-2-predictor-7dfc8cb68-tbrt7

Scheduled

Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7dfc8cb68-tbrt7 to ip-10-0-128-241.ec2.internal

kserve-ci-e2e-test

ensemble-graph-8493d-5fc7d65d87-6gpbl

Scheduled

Successfully assigned kserve-ci-e2e-test/ensemble-graph-8493d-5fc7d65d87-6gpbl to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

error-404-isvc-54da1-predictor-bf96fd7c-p48j9

Scheduled

Successfully assigned kserve-ci-e2e-test/error-404-isvc-54da1-predictor-bf96fd7c-p48j9 to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

success-200-isvc-d60f9-predictor-54b667d74d-nv5ff

Scheduled

Successfully assigned kserve-ci-e2e-test/success-200-isvc-d60f9-predictor-54b667d74d-nv5ff to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

switch-graph-d9a0e-5c6d76d677-vlrj4

Scheduled

Successfully assigned kserve-ci-e2e-test/switch-graph-d9a0e-5c6d76d677-vlrj4 to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

ensemble-graph-ed576-6d6fdb76b5-86vh5

Scheduled

Successfully assigned kserve-ci-e2e-test/ensemble-graph-ed576-6d6fdb76b5-86vh5 to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

success-200-isvc-c5b1b-predictor-5854f8b788-s9zzw

Scheduled

Successfully assigned kserve-ci-e2e-test/success-200-isvc-c5b1b-predictor-5854f8b788-s9zzw to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

error-404-isvc-8e251-predictor-64c47d4b4f-xj88b

Scheduled

Successfully assigned kserve-ci-e2e-test/error-404-isvc-8e251-predictor-64c47d4b4f-xj88b to ip-10-0-134-233.ec2.internal

kserve-ci-e2e-test

deployment-controller

isvc-sklearn-graph-2-predictor

ScalingReplicaSet

Scaled up replica set isvc-sklearn-graph-2-predictor-7dfc8cb68 from 0 to 1

kserve-ci-e2e-test

kubelet

isvc-sklearn-graph-1-predictor-5466bc54c4-p8jst

Pulling

Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:316603f897d12241b3c676dc4b9dc4be67d3abbf471c031d81dabf3287d0fc8d"

kserve-ci-e2e-test

replicaset-controller

isvc-sklearn-graph-2-predictor-7dfc8cb68

SuccessfulCreate

Created pod: isvc-sklearn-graph-2-predictor-7dfc8cb68-tbrt7

kserve-ci-e2e-test

kubelet

error-404-isvc-d60f9-predictor-bcfc89bfc-f6jf6

Pulling

Pulling image "quay.io/opendatahub/error-404-isvc:odh-pr-1426"

kserve-ci-e2e-test

multus

isvc-xgboost-graph-predictor-784cb989b8-4c5tl

AddedInterface

Add eth0 [10.132.0.32/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

isvc-xgboost-graph-predictor-784cb989b8-4c5tl

Pulling

Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:316603f897d12241b3c676dc4b9dc4be67d3abbf471c031d81dabf3287d0fc8d"

kserve-ci-e2e-test

v1beta1Controllers

isvc-sklearn-graph-1

UpdateFailed

Failed to update status for InferenceService "isvc-sklearn-graph-1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-1": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

deployment-controller

isvc-xgboost-graph-predictor

ScalingReplicaSet

Scaled up replica set isvc-xgboost-graph-predictor-784cb989b8 from 0 to 1

kserve-ci-e2e-test

v1beta1Controllers

isvc-sklearn-graph-1

InternalError

fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-1": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

replicaset-controller

success-200-isvc-d60f9-predictor-54b667d74d

SuccessfulCreate

Created pod: success-200-isvc-d60f9-predictor-54b667d74d-nv5ff

kserve-ci-e2e-test

deployment-controller

success-200-isvc-d60f9-predictor

ScalingReplicaSet

Scaled up replica set success-200-isvc-d60f9-predictor-54b667d74d from 0 to 1

kserve-ci-e2e-test

multus

error-404-isvc-d60f9-predictor-bcfc89bfc-f6jf6

AddedInterface

Add eth0 [10.133.0.28/23] from ovn-kubernetes

kserve-ci-e2e-test

deployment-controller

isvc-sklearn-graph-1-predictor

ScalingReplicaSet

Scaled up replica set isvc-sklearn-graph-1-predictor-5466bc54c4 from 0 to 1

kserve-ci-e2e-test

replicaset-controller

isvc-xgboost-graph-predictor-784cb989b8

SuccessfulCreate

Created pod: isvc-xgboost-graph-predictor-784cb989b8-4c5tl

kserve-ci-e2e-test

multus

success-200-isvc-d60f9-predictor-54b667d74d-nv5ff

AddedInterface

Add eth0 [10.133.0.27/23] from ovn-kubernetes

kserve-ci-e2e-test

multus

isvc-sklearn-graph-1-predictor-5466bc54c4-p8jst

AddedInterface

Add eth0 [10.132.0.31/23] from ovn-kubernetes

kserve-ci-e2e-test

replicaset-controller

isvc-sklearn-graph-1-predictor-5466bc54c4

SuccessfulCreate

Created pod: isvc-sklearn-graph-1-predictor-5466bc54c4-p8jst

kserve-ci-e2e-test

deployment-controller

error-404-isvc-d60f9-predictor

ScalingReplicaSet

Scaled up replica set error-404-isvc-d60f9-predictor-bcfc89bfc from 0 to 1

kserve-ci-e2e-test

kubelet

success-200-isvc-d60f9-predictor-54b667d74d-nv5ff

Pulling

Pulling image "quay.io/opendatahub/success-200-isvc:odh-pr-1426"

kserve-ci-e2e-test

replicaset-controller

error-404-isvc-d60f9-predictor-bcfc89bfc

SuccessfulCreate

Created pod: error-404-isvc-d60f9-predictor-bcfc89bfc-f6jf6

kserve-ci-e2e-test

kubelet

isvc-sklearn-graph-2-predictor-7dfc8cb68-tbrt7

Pulling

Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:316603f897d12241b3c676dc4b9dc4be67d3abbf471c031d81dabf3287d0fc8d"

kserve-ci-e2e-test

multus

isvc-sklearn-graph-2-predictor-7dfc8cb68-tbrt7

AddedInterface

Add eth0 [10.132.0.33/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

isvc-sklearn-graph-2-predictor-7dfc8cb68-tbrt7

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

isvc-xgboost-graph-predictor-784cb989b8-4c5tl

Pulled

Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:316603f897d12241b3c676dc4b9dc4be67d3abbf471c031d81dabf3287d0fc8d" in 4.108s (4.108s including waiting). Image size: 301288404 bytes.

kserve-ci-e2e-test

kubelet

isvc-sklearn-graph-1-predictor-5466bc54c4-p8jst

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

isvc-sklearn-graph-1-predictor-5466bc54c4-p8jst

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

isvc-sklearn-graph-2-predictor-7dfc8cb68-tbrt7

Pulled

Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:316603f897d12241b3c676dc4b9dc4be67d3abbf471c031d81dabf3287d0fc8d" in 3.897s (3.897s including waiting). Image size: 301288404 bytes.

kserve-ci-e2e-test

kubelet

isvc-sklearn-graph-2-predictor-7dfc8cb68-tbrt7

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

isvc-xgboost-graph-predictor-784cb989b8-4c5tl

Created

Created container: storage-initializer

kserve-ci-e2e-test

kubelet

isvc-sklearn-graph-1-predictor-5466bc54c4-p8jst

Pulled

Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:316603f897d12241b3c676dc4b9dc4be67d3abbf471c031d81dabf3287d0fc8d" in 4.365s (4.365s including waiting). Image size: 301288404 bytes.

kserve-ci-e2e-test

kubelet

isvc-xgboost-graph-predictor-784cb989b8-4c5tl

Started

Started container storage-initializer

kserve-ci-e2e-test

kubelet

isvc-xgboost-graph-predictor-784cb989b8-4c5tl

Pulling

Pulling image "kserve/xgbserver:latest"

kserve-ci-e2e-test

kubelet

isvc-sklearn-graph-1-predictor-5466bc54c4-p8jst

Pulling

Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1426"

kserve-ci-e2e-test

kubelet

isvc-sklearn-graph-2-predictor-7dfc8cb68-tbrt7

Pulling

Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1426"

kserve-ci-e2e-test

kubelet

error-404-isvc-d60f9-predictor-bcfc89bfc-f6jf6

Created

Created container: kserve-container

kserve-ci-e2e-test

kubelet

error-404-isvc-d60f9-predictor-bcfc89bfc-f6jf6

Started

Started container kserve-container

kserve-ci-e2e-test

kubelet

success-200-isvc-d60f9-predictor-54b667d74d-nv5ff

Started

Started container kserve-container

kserve-ci-e2e-test

kubelet

success-200-isvc-d60f9-predictor-54b667d74d-nv5ff

Created

Created container: kserve-container

kserve-ci-e2e-test

kubelet

success-200-isvc-d60f9-predictor-54b667d74d-nv5ff

Pulled

Successfully pulled image "quay.io/opendatahub/success-200-isvc:odh-pr-1426" in 15.122s (15.122s including waiting). Image size: 1332348033 bytes.

kserve-ci-e2e-test

kubelet

error-404-isvc-d60f9-predictor-bcfc89bfc-f6jf6

Pulled

Successfully pulled image "quay.io/opendatahub/error-404-isvc:odh-pr-1426" in 14.877s (14.877s including waiting). Image size: 1332236408 bytes.

kserve-ci-e2e-test

kubelet

isvc-xgboost-graph-predictor-784cb989b8-4c5tl

Created

Created container: kserve-container

kserve-ci-e2e-test

kubelet

isvc-sklearn-graph-1-predictor-5466bc54c4-p8jst

Started

Started container kserve-container

kserve-ci-e2e-test

kubelet

isvc-sklearn-graph-2-predictor-7dfc8cb68-tbrt7

Started

Started container kserve-container

kserve-ci-e2e-test

kubelet

isvc-sklearn-graph-2-predictor-7dfc8cb68-tbrt7

Created

Created container: kserve-container

kserve-ci-e2e-test

kubelet

isvc-sklearn-graph-2-predictor-7dfc8cb68-tbrt7

Pulled

Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1426" in 28.648s (28.648s including waiting). Image size: 1560702918 bytes.

kserve-ci-e2e-test

kubelet

isvc-sklearn-graph-1-predictor-5466bc54c4-p8jst

Created

Created container: kserve-container

kserve-ci-e2e-test

kubelet

isvc-xgboost-graph-predictor-784cb989b8-4c5tl

Pulled

Successfully pulled image "kserve/xgbserver:latest" in 28.759s (28.759s including waiting). Image size: 1306229499 bytes.

kserve-ci-e2e-test

kubelet

isvc-xgboost-graph-predictor-784cb989b8-4c5tl

Started

Started container kserve-container

kserve-ci-e2e-test

kubelet

isvc-sklearn-graph-1-predictor-5466bc54c4-p8jst

Pulled

Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1426" in 28.637s (28.637s including waiting). Image size: 1560702918 bytes.
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

isvc-xgboost-graph-predictor

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

isvc-xgboost-graph-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-d60f9-predictor

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-d60f9-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

isvc-sklearn-graph-2-predictor

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-d60f9-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-d60f9-predictor

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

isvc-sklearn-graph-2-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x6)

kserve-ci-e2e-test

kubelet

success-200-isvc-d60f9-predictor-54b667d74d-nv5ff

Unhealthy

Readiness probe failed: dial tcp 10.133.0.27:8080: connect: connection refused
(x6)

kserve-ci-e2e-test

kubelet

error-404-isvc-d60f9-predictor-bcfc89bfc-f6jf6

Unhealthy

Readiness probe failed: dial tcp 10.133.0.28:8080: connect: connection refused
(x5)

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-d60f9

VirtualServiceCRDNotFound

Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
(x5)

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-d60f9

VirtualServiceCRDNotFound

Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-d60f9

InferenceServiceReady

InferenceService [error-404-isvc-d60f9] is Ready

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-d60f9

InferenceServiceReady

InferenceService [success-200-isvc-d60f9] is Ready
(x5)

kserve-ci-e2e-test

horizontal-pod-autoscaler

isvc-sklearn-graph-1-predictor

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x5)

kserve-ci-e2e-test

horizontal-pod-autoscaler

isvc-sklearn-graph-1-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kserve-ci-e2e-test

InferenceGraphController

switch-graph-d60f9

InferenceGraphReady

InferenceGraph [switch-graph-d60f9] is Ready

kserve-ci-e2e-test

deployment-controller

switch-graph-d60f9

ScalingReplicaSet

Scaled up replica set switch-graph-d60f9-db987dd58 from 0 to 1

kserve-ci-e2e-test

replicaset-controller

switch-graph-d60f9-db987dd58

SuccessfulCreate

Created pod: switch-graph-d60f9-db987dd58-m2b28

kserve-ci-e2e-test

kubelet

switch-graph-d60f9-db987dd58-m2b28

FailedMount

MountVolume.SetUp failed for volume "proxy-tls" : secret "switch-graph-d60f9-serving-cert" not found

kserve-ci-e2e-test

kubelet

switch-graph-d60f9-db987dd58-m2b28

Pulling

Pulling image "quay.io/opendatahub/kserve-router@sha256:dd92bbb6db070213e4821d314fe0d4d7707048c2ffbd06bddadfaae5e74b343a"

kserve-ci-e2e-test

multus

switch-graph-d60f9-db987dd58-m2b28

AddedInterface

Add eth0 [10.133.0.29/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

switch-graph-d60f9-db987dd58-m2b28

Created

Created container: switch-graph-d60f9

kserve-ci-e2e-test

kubelet

switch-graph-d60f9-db987dd58-m2b28

Pulled

Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:dd92bbb6db070213e4821d314fe0d4d7707048c2ffbd06bddadfaae5e74b343a" in 2.55s (2.55s including waiting). Image size: 216233593 bytes.

kserve-ci-e2e-test

kubelet

switch-graph-d60f9-db987dd58-m2b28

Started

Started container switch-graph-d60f9
(x7)

kserve-ci-e2e-test

kubelet

isvc-xgboost-graph-predictor-784cb989b8-4c5tl

Unhealthy

Readiness probe failed: dial tcp 10.132.0.32:8080: connect: connection refused

kserve-ci-e2e-test

kubelet

success-200-isvc-d60f9-predictor-54b667d74d-nv5ff

Killing

Stopping container kserve-container

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-8e251

InternalError

fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-8e251": the object has been modified; please apply your changes to the latest version and try again
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-d60f9-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

kserve-ci-e2e-test

replicaset-controller

success-200-isvc-8e251-predictor-59cb876bdd

SuccessfulCreate

Created pod: success-200-isvc-8e251-predictor-59cb876bdd-dx2p4
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-d60f9-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-d60f9-predictor

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

kserve-ci-e2e-test

kubelet

success-200-isvc-8e251-predictor-59cb876bdd-dx2p4

Pulled

Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1426" already present on machine

kserve-ci-e2e-test

multus

success-200-isvc-8e251-predictor-59cb876bdd-dx2p4

AddedInterface

Add eth0 [10.133.0.30/23] from ovn-kubernetes

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-8e251

UpdateFailed

Failed to update status for InferenceService "error-404-isvc-8e251": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-8e251": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

deployment-controller

success-200-isvc-8e251-predictor

ScalingReplicaSet

Scaled up replica set success-200-isvc-8e251-predictor-59cb876bdd from 0 to 1

kserve-ci-e2e-test

kubelet

error-404-isvc-d60f9-predictor-bcfc89bfc-f6jf6

Killing

Stopping container kserve-container

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-8e251

InternalError

fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-8e251": the object has been modified; please apply your changes to the latest version and try again
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-d60f9-predictor

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

kserve-ci-e2e-test

kubelet

switch-graph-d60f9-db987dd58-m2b28

Killing

Stopping container switch-graph-d60f9

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-8e251

UpdateFailed

Failed to update status for InferenceService "success-200-isvc-8e251": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-8e251": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

deployment-controller

error-404-isvc-8e251-predictor

ScalingReplicaSet

Scaled up replica set error-404-isvc-8e251-predictor-64c47d4b4f from 0 to 1

kserve-ci-e2e-test

replicaset-controller

error-404-isvc-8e251-predictor-64c47d4b4f

SuccessfulCreate

Created pod: error-404-isvc-8e251-predictor-64c47d4b4f-xj88b

kserve-ci-e2e-test

kubelet

success-200-isvc-8e251-predictor-59cb876bdd-dx2p4

Created

Created container: kserve-container

kserve-ci-e2e-test

kubelet

error-404-isvc-8e251-predictor-64c47d4b4f-xj88b

Pulled

Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1426" already present on machine

kserve-ci-e2e-test

kubelet

success-200-isvc-8e251-predictor-59cb876bdd-dx2p4

Started

Started container kserve-container

kserve-ci-e2e-test

multus

error-404-isvc-8e251-predictor-64c47d4b4f-xj88b

AddedInterface

Add eth0 [10.133.0.31/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

error-404-isvc-8e251-predictor-64c47d4b4f-xj88b

Started

Started container kserve-container

kserve-ci-e2e-test

kubelet

error-404-isvc-8e251-predictor-64c47d4b4f-xj88b

Created

Created container: kserve-container
(x9)

kserve-ci-e2e-test

v1beta1Controllers

isvc-xgboost-graph

VirtualServiceCRDNotFound

Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.

kserve-ci-e2e-test

v1beta1Controllers

isvc-xgboost-graph

InferenceServiceReady

InferenceService [isvc-xgboost-graph] is Ready
(x8)

kserve-ci-e2e-test

kubelet

isvc-sklearn-graph-2-predictor-7dfc8cb68-tbrt7

Unhealthy

Readiness probe failed: dial tcp 10.132.0.33:8080: connect: connection refused
(x8)

kserve-ci-e2e-test

kubelet

isvc-sklearn-graph-1-predictor-5466bc54c4-p8jst

Unhealthy

Readiness probe failed: dial tcp 10.132.0.31:8080: connect: connection refused

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-8e251-predictor

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-8e251-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kserve-ci-e2e-test

v1beta1Controllers

isvc-sklearn-graph-2

InferenceServiceReady

InferenceService [isvc-sklearn-graph-2] is Ready

kserve-ci-e2e-test

v1beta1Controllers

isvc-sklearn-graph-1

InferenceServiceReady

InferenceService [isvc-sklearn-graph-1] is Ready
(x9)

kserve-ci-e2e-test

v1beta1Controllers

isvc-sklearn-graph-2

VirtualServiceCRDNotFound

Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
(x9)

kserve-ci-e2e-test

v1beta1Controllers

isvc-sklearn-graph-1

VirtualServiceCRDNotFound

Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
(x6)

kserve-ci-e2e-test

kubelet

switch-graph-d60f9-db987dd58-m2b28

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-8e251-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-8e251-predictor

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kserve-ci-e2e-test

InferenceGraphController

model-chainer

InferenceGraphReady

InferenceGraph [model-chainer] is Ready

kserve-ci-e2e-test

deployment-controller

model-chainer

ScalingReplicaSet

Scaled up replica set model-chainer-884b78d8f from 0 to 1

kserve-ci-e2e-test

kubelet

model-chainer-884b78d8f-rz4zw

FailedMount

MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-serving-cert" not found

kserve-ci-e2e-test

replicaset-controller

model-chainer-884b78d8f

SuccessfulCreate

Created pod: model-chainer-884b78d8f-rz4zw

kserve-ci-e2e-test

kubelet

model-chainer-884b78d8f-rz4zw

Pulled

Container image "quay.io/opendatahub/kserve-router@sha256:dd92bbb6db070213e4821d314fe0d4d7707048c2ffbd06bddadfaae5e74b343a" already present on machine

kserve-ci-e2e-test

multus

model-chainer-884b78d8f-rz4zw

AddedInterface

Add eth0 [10.133.0.32/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

model-chainer-884b78d8f-rz4zw

Created

Created container: model-chainer

kserve-ci-e2e-test

kubelet

model-chainer-884b78d8f-rz4zw

Started

Started container model-chainer
(x6)

kserve-ci-e2e-test

kubelet

success-200-isvc-8e251-predictor-59cb876bdd-dx2p4

Unhealthy

Readiness probe failed: dial tcp 10.133.0.30:8080: connect: connection refused
(x6)

kserve-ci-e2e-test

kubelet

error-404-isvc-8e251-predictor-64c47d4b4f-xj88b

Unhealthy

Readiness probe failed: dial tcp 10.133.0.31:8080: connect: connection refused
(x6)

kserve-ci-e2e-test

horizontal-pod-autoscaler

isvc-sklearn-graph-2-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x6)

kserve-ci-e2e-test

horizontal-pod-autoscaler

isvc-xgboost-graph-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x6)

kserve-ci-e2e-test

horizontal-pod-autoscaler

isvc-xgboost-graph-predictor

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x6)

kserve-ci-e2e-test

horizontal-pod-autoscaler

isvc-sklearn-graph-2-predictor

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x4)

kserve-ci-e2e-test

horizontal-pod-autoscaler

isvc-sklearn-graph-1-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x4)

kserve-ci-e2e-test

horizontal-pod-autoscaler

isvc-sklearn-graph-1-predictor

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

kserve-ci-e2e-test

kubelet

error-404-isvc-f6449-predictor-5db958dfd5-jktbg

Created

Created container: kserve-container

kserve-ci-e2e-test

kubelet

success-200-isvc-f6449-predictor-67b49d9f68-jvnsv

Pulled

Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1426" already present on machine

kserve-ci-e2e-test

kubelet

isvc-sklearn-graph-1-predictor-5466bc54c4-p8jst

Killing

Stopping container kserve-container

kserve-ci-e2e-test

multus

success-200-isvc-f6449-predictor-67b49d9f68-jvnsv

AddedInterface

Add eth0 [10.133.0.33/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

isvc-xgboost-graph-predictor-784cb989b8-4c5tl

Killing

Stopping container kserve-container

kserve-ci-e2e-test

kubelet

model-chainer-884b78d8f-rz4zw

Killing

Stopping container model-chainer

kserve-ci-e2e-test

kubelet

success-200-isvc-f6449-predictor-67b49d9f68-jvnsv

Created

Created container: kserve-container

kserve-ci-e2e-test

kubelet

success-200-isvc-f6449-predictor-67b49d9f68-jvnsv

Started

Started container kserve-container

kserve-ci-e2e-test

deployment-controller

error-404-isvc-f6449-predictor

ScalingReplicaSet

Scaled up replica set error-404-isvc-f6449-predictor-5db958dfd5 from 0 to 1

kserve-ci-e2e-test

kubelet

isvc-sklearn-graph-2-predictor-7dfc8cb68-tbrt7

Killing

Stopping container kserve-container

kserve-ci-e2e-test

replicaset-controller

error-404-isvc-f6449-predictor-5db958dfd5

SuccessfulCreate

Created pod: error-404-isvc-f6449-predictor-5db958dfd5-jktbg

kserve-ci-e2e-test

replicaset-controller

success-200-isvc-f6449-predictor-67b49d9f68

SuccessfulCreate

Created pod: success-200-isvc-f6449-predictor-67b49d9f68-jvnsv

kserve-ci-e2e-test

multus

error-404-isvc-f6449-predictor-5db958dfd5-jktbg

AddedInterface

Add eth0 [10.133.0.34/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

error-404-isvc-f6449-predictor-5db958dfd5-jktbg

Pulled

Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1426" already present on machine

kserve-ci-e2e-test

deployment-controller

success-200-isvc-f6449-predictor

ScalingReplicaSet

Scaled up replica set success-200-isvc-f6449-predictor-67b49d9f68 from 0 to 1

kserve-ci-e2e-test

kubelet

error-404-isvc-f6449-predictor-5db958dfd5-jktbg

Started

Started container kserve-container

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-8e251

InferenceServiceReady

InferenceService [success-200-isvc-8e251] is Ready
(x5)

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-8e251

VirtualServiceCRDNotFound

Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-8e251

InferenceServiceReady

InferenceService [error-404-isvc-8e251] is Ready
(x5)

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-8e251

VirtualServiceCRDNotFound

Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.

kserve-ci-e2e-test

deployment-controller

switch-graph-8e251

ScalingReplicaSet

Scaled up replica set switch-graph-8e251-5896fcfc4c from 0 to 1

kserve-ci-e2e-test

replicaset-controller

switch-graph-8e251-5896fcfc4c

SuccessfulCreate

Created pod: switch-graph-8e251-5896fcfc4c-jr8mf

kserve-ci-e2e-test

InferenceGraphController

switch-graph-8e251

InferenceGraphReady

InferenceGraph [switch-graph-8e251] is Ready
(x2)

kserve-ci-e2e-test

kubelet

switch-graph-8e251-5896fcfc4c-jr8mf

FailedMount

MountVolume.SetUp failed for volume "proxy-tls" : secret "switch-graph-8e251-serving-cert" not found

kserve-ci-e2e-test

multus

switch-graph-8e251-5896fcfc4c-jr8mf

AddedInterface

Add eth0 [10.133.0.35/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

switch-graph-8e251-5896fcfc4c-jr8mf

Pulled

Container image "quay.io/opendatahub/kserve-router@sha256:dd92bbb6db070213e4821d314fe0d4d7707048c2ffbd06bddadfaae5e74b343a" already present on machine

kserve-ci-e2e-test

kubelet

switch-graph-8e251-5896fcfc4c-jr8mf

Created

Created container: switch-graph-8e251

kserve-ci-e2e-test

kubelet

switch-graph-8e251-5896fcfc4c-jr8mf

Started

Started container switch-graph-8e251
(x4)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-8e251-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x4)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-8e251-predictor

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x6)

kserve-ci-e2e-test

kubelet

model-chainer-884b78d8f-rz4zw

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503
(x4)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-8e251-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x4)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-8e251-predictor

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x6)

kserve-ci-e2e-test

kubelet

error-404-isvc-f6449-predictor-5db958dfd5-jktbg

Unhealthy

Readiness probe failed: dial tcp 10.133.0.34:8080: connect: connection refused
(x6)

kserve-ci-e2e-test

kubelet

success-200-isvc-f6449-predictor-67b49d9f68-jvnsv

Unhealthy

Readiness probe failed: dial tcp 10.133.0.33:8080: connect: connection refused
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-f6449-predictor

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-f6449-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

switch-graph-8e251

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

switch-graph-8e251

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x5)

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-f6449

VirtualServiceCRDNotFound

Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-f6449

InferenceServiceReady

InferenceService [error-404-isvc-f6449] is Ready

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-f6449

InferenceServiceReady

InferenceService [success-200-isvc-f6449] is Ready
(x5)

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-f6449

VirtualServiceCRDNotFound

Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
(x4)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-f6449-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x4)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-f6449-predictor

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kserve-ci-e2e-test

replicaset-controller

sequence-graph-f6449-6ff9b5b46f

SuccessfulCreate

Created pod: sequence-graph-f6449-6ff9b5b46f-jl9sk

kserve-ci-e2e-test

deployment-controller

sequence-graph-f6449

ScalingReplicaSet

Scaled up replica set sequence-graph-f6449-6ff9b5b46f from 0 to 1

kserve-ci-e2e-test

InferenceGraphController

sequence-graph-f6449

InferenceGraphReady

InferenceGraph [sequence-graph-f6449] is Ready

kserve-ci-e2e-test

multus

sequence-graph-f6449-6ff9b5b46f-jl9sk

AddedInterface

Add eth0 [10.133.0.36/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

sequence-graph-f6449-6ff9b5b46f-jl9sk

Pulled

Container image "quay.io/opendatahub/kserve-router@sha256:dd92bbb6db070213e4821d314fe0d4d7707048c2ffbd06bddadfaae5e74b343a" already present on machine

kserve-ci-e2e-test

kubelet

sequence-graph-f6449-6ff9b5b46f-jl9sk

Created

Created container: sequence-graph-f6449

kserve-ci-e2e-test

kubelet

sequence-graph-f6449-6ff9b5b46f-jl9sk

Started

Started container sequence-graph-f6449

kserve-ci-e2e-test

horizontal-pod-autoscaler

sequence-graph-f6449

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kserve-ci-e2e-test

horizontal-pod-autoscaler

sequence-graph-f6449

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-f6449-predictor

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-f6449-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-f6449-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-f6449-predictor

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

kserve-ci-e2e-test

horizontal-pod-autoscaler

sequence-graph-f6449

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

kserve-ci-e2e-test

horizontal-pod-autoscaler

sequence-graph-f6449

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

kserve-ci-e2e-test

kubelet

success-200-isvc-8e251-predictor-59cb876bdd-dx2p4

Killing

Stopping container kserve-container

kserve-ci-e2e-test

kubelet

error-404-isvc-8e251-predictor-64c47d4b4f-xj88b

Killing

Stopping container kserve-container

kserve-ci-e2e-test

kubelet

switch-graph-8e251-5896fcfc4c-jr8mf

Killing

Stopping container switch-graph-8e251

kserve-ci-e2e-test

deployment-controller

success-200-isvc-8493d-predictor

ScalingReplicaSet

Scaled up replica set success-200-isvc-8493d-predictor-77b4959486 from 0 to 1

kserve-ci-e2e-test

kubelet

success-200-isvc-8493d-predictor-77b4959486-7d762

Pulled

Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1426" already present on machine

kserve-ci-e2e-test

kubelet

success-200-isvc-8493d-predictor-77b4959486-7d762

Created

Created container: kserve-container

kserve-ci-e2e-test

kubelet

error-404-isvc-8493d-predictor-788f857f9f-wlsz5

Started

Started container kserve-container

kserve-ci-e2e-test

multus

success-200-isvc-8493d-predictor-77b4959486-7d762

AddedInterface

Add eth0 [10.133.0.37/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

success-200-isvc-8493d-predictor-77b4959486-7d762

Started

Started container kserve-container

kserve-ci-e2e-test

replicaset-controller

success-200-isvc-8493d-predictor-77b4959486

SuccessfulCreate

Created pod: success-200-isvc-8493d-predictor-77b4959486-7d762

kserve-ci-e2e-test

replicaset-controller

error-404-isvc-8493d-predictor-788f857f9f

SuccessfulCreate

Created pod: error-404-isvc-8493d-predictor-788f857f9f-wlsz5

kserve-ci-e2e-test

deployment-controller

error-404-isvc-8493d-predictor

ScalingReplicaSet

Scaled up replica set error-404-isvc-8493d-predictor-788f857f9f from 0 to 1

kserve-ci-e2e-test

kubelet

error-404-isvc-8493d-predictor-788f857f9f-wlsz5

Pulled

Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1426" already present on machine

kserve-ci-e2e-test

multus

error-404-isvc-8493d-predictor-788f857f9f-wlsz5

AddedInterface

Add eth0 [10.133.0.38/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

error-404-isvc-8493d-predictor-788f857f9f-wlsz5

Created

Created container: kserve-container
(x6)

kserve-ci-e2e-test

kubelet

switch-graph-8e251-5896fcfc4c-jr8mf

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-8493d-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-8493d-predictor

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-8493d-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-8493d-predictor

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kserve-ci-e2e-test

kubelet

error-404-isvc-5e299-predictor-75c4868959-6fvvq

Pulled

Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1426" already present on machine

kserve-ci-e2e-test

kubelet

error-404-isvc-5e299-predictor-75c4868959-6fvvq

Created

Created container: kserve-container

kserve-ci-e2e-test

multus

error-404-isvc-5e299-predictor-75c4868959-6fvvq

AddedInterface

Add eth0 [10.133.0.40/23] from ovn-kubernetes

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-5e299

UpdateFailed

Failed to update status for InferenceService "success-200-isvc-5e299": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-5e299": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

deployment-controller

error-404-isvc-5e299-predictor

ScalingReplicaSet

Scaled up replica set error-404-isvc-5e299-predictor-75c4868959 from 0 to 1

kserve-ci-e2e-test

kubelet

success-200-isvc-f6449-predictor-67b49d9f68-jvnsv

Killing

Stopping container kserve-container

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-5e299

InternalError

fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-5e299": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-5e299

UpdateFailed

Failed to update status for InferenceService "error-404-isvc-5e299": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-5e299": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

kubelet

sequence-graph-f6449-6ff9b5b46f-jl9sk

Killing

Stopping container sequence-graph-f6449

kserve-ci-e2e-test

kubelet

error-404-isvc-f6449-predictor-5db958dfd5-jktbg

Killing

Stopping container kserve-container

kserve-ci-e2e-test

deployment-controller

success-200-isvc-5e299-predictor

ScalingReplicaSet

Scaled up replica set success-200-isvc-5e299-predictor-7d5dc4956b from 0 to 1

kserve-ci-e2e-test

multus

success-200-isvc-5e299-predictor-7d5dc4956b-d27ll

AddedInterface

Add eth0 [10.133.0.39/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

success-200-isvc-5e299-predictor-7d5dc4956b-d27ll

Pulled

Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1426" already present on machine

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-5e299

InternalError

fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-5e299": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

kubelet

error-404-isvc-5e299-predictor-75c4868959-6fvvq

Started

Started container kserve-container

kserve-ci-e2e-test

replicaset-controller

success-200-isvc-5e299-predictor-7d5dc4956b

SuccessfulCreate

Created pod: success-200-isvc-5e299-predictor-7d5dc4956b-d27ll

kserve-ci-e2e-test

kubelet

success-200-isvc-5e299-predictor-7d5dc4956b-d27ll

Started

Started container kserve-container

kserve-ci-e2e-test

kubelet

success-200-isvc-5e299-predictor-7d5dc4956b-d27ll

Created

Created container: kserve-container

kserve-ci-e2e-test

replicaset-controller

error-404-isvc-5e299-predictor-75c4868959

SuccessfulCreate

Created pod: error-404-isvc-5e299-predictor-75c4868959-6fvvq

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-8493d

InferenceServiceReady

InferenceService [error-404-isvc-8493d] is Ready

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-8493d

InferenceServiceReady

InferenceService [success-200-isvc-8493d] is Ready
(x5)

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-8493d

VirtualServiceCRDNotFound

Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
(x6)

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-8493d

VirtualServiceCRDNotFound

Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-5e299-predictor

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-5e299-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kserve-ci-e2e-test

replicaset-controller

ensemble-graph-8493d-5fc7d65d87

SuccessfulCreate

Created pod: ensemble-graph-8493d-5fc7d65d87-6gpbl

kserve-ci-e2e-test

deployment-controller

ensemble-graph-8493d

ScalingReplicaSet

Scaled up replica set ensemble-graph-8493d-5fc7d65d87 from 0 to 1

kserve-ci-e2e-test

kubelet

ensemble-graph-8493d-5fc7d65d87-6gpbl

Started

Started container ensemble-graph-8493d

kserve-ci-e2e-test

InferenceGraphController

ensemble-graph-8493d

InferenceGraphReady

InferenceGraph [ensemble-graph-8493d] is Ready

kserve-ci-e2e-test

kubelet

ensemble-graph-8493d-5fc7d65d87-6gpbl

Created

Created container: ensemble-graph-8493d

kserve-ci-e2e-test

kubelet

ensemble-graph-8493d-5fc7d65d87-6gpbl

Pulled

Container image "quay.io/opendatahub/kserve-router@sha256:dd92bbb6db070213e4821d314fe0d4d7707048c2ffbd06bddadfaae5e74b343a" already present on machine

kserve-ci-e2e-test

multus

ensemble-graph-8493d-5fc7d65d87-6gpbl

AddedInterface

Add eth0 [10.133.0.41/23] from ovn-kubernetes
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-8493d-predictor

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-8493d-predictor

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-8493d-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-8493d-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x6)

kserve-ci-e2e-test

kubelet

sequence-graph-f6449-6ff9b5b46f-jl9sk

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-5e299-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-5e299-predictor

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kserve-ci-e2e-test

deployment-controller

error-404-isvc-ed576-predictor

ScalingReplicaSet

Scaled up replica set error-404-isvc-ed576-predictor-65c7fd45bf from 0 to 1

kserve-ci-e2e-test

kubelet

ensemble-graph-8493d-5fc7d65d87-6gpbl

Killing

Stopping container ensemble-graph-8493d

kserve-ci-e2e-test

deployment-controller

success-200-isvc-ed576-predictor

ScalingReplicaSet

Scaled up replica set success-200-isvc-ed576-predictor-78dc55c9fb from 0 to 1

kserve-ci-e2e-test

replicaset-controller

success-200-isvc-ed576-predictor-78dc55c9fb

SuccessfulCreate

Created pod: success-200-isvc-ed576-predictor-78dc55c9fb-zr27l

kserve-ci-e2e-test

kubelet

success-200-isvc-ed576-predictor-78dc55c9fb-zr27l

Started

Started container kserve-container

kserve-ci-e2e-test

kubelet

success-200-isvc-ed576-predictor-78dc55c9fb-zr27l

Created

Created container: kserve-container

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-ed576

UpdateFailed

Failed to update status for InferenceService "success-200-isvc-ed576": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-ed576": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

kubelet

error-404-isvc-ed576-predictor-65c7fd45bf-jfsc8

Started

Started container kserve-container

kserve-ci-e2e-test

kubelet

error-404-isvc-ed576-predictor-65c7fd45bf-jfsc8

Pulled

Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1426" already present on machine

kserve-ci-e2e-test

multus

error-404-isvc-ed576-predictor-65c7fd45bf-jfsc8

AddedInterface

Add eth0 [10.133.0.43/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

success-200-isvc-8493d-predictor-77b4959486-7d762

Killing

Stopping container kserve-container

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-ed576

InternalError

fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-ed576": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

kubelet

success-200-isvc-ed576-predictor-78dc55c9fb-zr27l

Pulled

Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1426" already present on machine

kserve-ci-e2e-test

multus

success-200-isvc-ed576-predictor-78dc55c9fb-zr27l

AddedInterface

Add eth0 [10.133.0.42/23] from ovn-kubernetes

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-ed576

InternalError

fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-ed576": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

kubelet

error-404-isvc-8493d-predictor-788f857f9f-wlsz5

Killing

Stopping container kserve-container

kserve-ci-e2e-test

replicaset-controller

error-404-isvc-ed576-predictor-65c7fd45bf

SuccessfulCreate

Created pod: error-404-isvc-ed576-predictor-65c7fd45bf-jfsc8

kserve-ci-e2e-test

kubelet

error-404-isvc-ed576-predictor-65c7fd45bf-jfsc8

Created

Created container: kserve-container

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-ed576

UpdateFailed

Failed to update status for InferenceService "error-404-isvc-ed576": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-ed576": the object has been modified; please apply your changes to the latest version and try again
(x7)

kserve-ci-e2e-test

kubelet

success-200-isvc-8493d-predictor-77b4959486-7d762

Unhealthy

Readiness probe failed: dial tcp 10.133.0.37:8080: connect: connection refused
(x7)

kserve-ci-e2e-test

kubelet

error-404-isvc-8493d-predictor-788f857f9f-wlsz5

Unhealthy

Readiness probe failed: dial tcp 10.133.0.38:8080: connect: connection refused
(x6)

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-5e299

VirtualServiceCRDNotFound

Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-5e299

InferenceServiceReady

InferenceService [error-404-isvc-5e299] is Ready

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-5e299

InferenceServiceReady

InferenceService [success-200-isvc-5e299] is Ready
(x5)

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-5e299

VirtualServiceCRDNotFound

Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
(x6)

kserve-ci-e2e-test

kubelet

ensemble-graph-8493d-5fc7d65d87-6gpbl

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-ed576-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-ed576-predictor

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-ed576-predictor

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-ed576-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kserve-ci-e2e-test

InferenceGraphController

sequence-graph-5e299

InternalError

fails to update InferenceGraph status: Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "sequence-graph-5e299": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

InferenceGraphController

sequence-graph-5e299

UpdateFailed

Failed to update status for InferenceGraph "sequence-graph-5e299": Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "sequence-graph-5e299": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

InferenceGraphController

sequence-graph-5e299

InferenceGraphReady

InferenceGraph [sequence-graph-5e299] is Ready

kserve-ci-e2e-test

deployment-controller

sequence-graph-5e299

ScalingReplicaSet

Scaled up replica set sequence-graph-5e299-6f5777484f from 0 to 1

kserve-ci-e2e-test

replicaset-controller

sequence-graph-5e299-6f5777484f

SuccessfulCreate

Created pod: sequence-graph-5e299-6f5777484f-5l654
(x2)

kserve-ci-e2e-test

kubelet

sequence-graph-5e299-6f5777484f-5l654

FailedMount

MountVolume.SetUp failed for volume "proxy-tls" : secret "sequence-graph-5e299-serving-cert" not found

kserve-ci-e2e-test

kubelet

sequence-graph-5e299-6f5777484f-5l654

Started

Started container sequence-graph-5e299

kserve-ci-e2e-test

multus

sequence-graph-5e299-6f5777484f-5l654

AddedInterface

Add eth0 [10.133.0.44/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

sequence-graph-5e299-6f5777484f-5l654

Created

Created container: sequence-graph-5e299

kserve-ci-e2e-test

kubelet

sequence-graph-5e299-6f5777484f-5l654

Pulled

Container image "quay.io/opendatahub/kserve-router@sha256:dd92bbb6db070213e4821d314fe0d4d7707048c2ffbd06bddadfaae5e74b343a" already present on machine
(x6)

kserve-ci-e2e-test

kubelet

success-200-isvc-ed576-predictor-78dc55c9fb-zr27l

Unhealthy

Readiness probe failed: dial tcp 10.133.0.42:8080: connect: connection refused
(x6)

kserve-ci-e2e-test

kubelet

error-404-isvc-ed576-predictor-65c7fd45bf-jfsc8

Unhealthy

Readiness probe failed: dial tcp 10.133.0.43:8080: connect: connection refused
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-5e299-predictor

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x4)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-5e299-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-5e299-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x4)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-5e299-predictor

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

kserve-ci-e2e-test

kubelet

success-200-isvc-5e299-predictor-7d5dc4956b-d27ll

Killing

Stopping container kserve-container

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-54da1

UpdateFailed

Failed to update status for InferenceService "error-404-isvc-54da1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-54da1": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-54da1

InternalError

fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-54da1": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

deployment-controller

success-200-isvc-54da1-predictor

ScalingReplicaSet

Scaled up replica set success-200-isvc-54da1-predictor-59846b8c76 from 0 to 1

kserve-ci-e2e-test

replicaset-controller

success-200-isvc-54da1-predictor-59846b8c76

SuccessfulCreate

Created pod: success-200-isvc-54da1-predictor-59846b8c76-rhmxv

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-54da1

InternalError

fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-54da1": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

kubelet

error-404-isvc-5e299-predictor-75c4868959-6fvvq

Killing

Stopping container kserve-container

kserve-ci-e2e-test

kubelet

success-200-isvc-54da1-predictor-59846b8c76-rhmxv

Started

Started container kserve-container

kserve-ci-e2e-test

kubelet

success-200-isvc-54da1-predictor-59846b8c76-rhmxv

Created

Created container: kserve-container

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-54da1

UpdateFailed

Failed to update status for InferenceService "success-200-isvc-54da1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-54da1": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

multus

error-404-isvc-54da1-predictor-bf96fd7c-p48j9

AddedInterface

Add eth0 [10.133.0.46/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

success-200-isvc-54da1-predictor-59846b8c76-rhmxv

Pulled

Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1426" already present on machine

kserve-ci-e2e-test

replicaset-controller

error-404-isvc-54da1-predictor-bf96fd7c

SuccessfulCreate

Created pod: error-404-isvc-54da1-predictor-bf96fd7c-p48j9

kserve-ci-e2e-test

kubelet

sequence-graph-5e299-6f5777484f-5l654

Killing

Stopping container sequence-graph-5e299

kserve-ci-e2e-test

kubelet

error-404-isvc-54da1-predictor-bf96fd7c-p48j9

Pulled

Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1426" already present on machine

kserve-ci-e2e-test

multus

success-200-isvc-54da1-predictor-59846b8c76-rhmxv

AddedInterface

Add eth0 [10.133.0.45/23] from ovn-kubernetes

kserve-ci-e2e-test

deployment-controller

error-404-isvc-54da1-predictor

ScalingReplicaSet

Scaled up replica set error-404-isvc-54da1-predictor-bf96fd7c from 0 to 1

kserve-ci-e2e-test

kubelet

error-404-isvc-54da1-predictor-bf96fd7c-p48j9

Created

Created container: kserve-container

kserve-ci-e2e-test

kubelet

error-404-isvc-54da1-predictor-bf96fd7c-p48j9

Started

Started container kserve-container
(x7)

kserve-ci-e2e-test

kubelet

success-200-isvc-5e299-predictor-7d5dc4956b-d27ll

Unhealthy

Readiness probe failed: dial tcp 10.133.0.39:8080: connect: connection refused
(x7)

kserve-ci-e2e-test

kubelet

error-404-isvc-5e299-predictor-75c4868959-6fvvq

Unhealthy

Readiness probe failed: dial tcp 10.133.0.40:8080: connect: connection refused
(x5)

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-ed576

VirtualServiceCRDNotFound

Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-ed576

InferenceServiceReady

InferenceService [error-404-isvc-ed576] is Ready
(x6)

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-ed576

VirtualServiceCRDNotFound

Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-ed576

InferenceServiceReady

InferenceService [success-200-isvc-ed576] is Ready

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-54da1-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-54da1-predictor

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-54da1-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-54da1-predictor

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kserve-ci-e2e-test

kubelet

ensemble-graph-ed576-6d6fdb76b5-86vh5

Started

Started container ensemble-graph-ed576

kserve-ci-e2e-test

kubelet

ensemble-graph-ed576-6d6fdb76b5-86vh5

Pulled

Container image "quay.io/opendatahub/kserve-router@sha256:dd92bbb6db070213e4821d314fe0d4d7707048c2ffbd06bddadfaae5e74b343a" already present on machine

kserve-ci-e2e-test

multus

ensemble-graph-ed576-6d6fdb76b5-86vh5

AddedInterface

Add eth0 [10.133.0.47/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

ensemble-graph-ed576-6d6fdb76b5-86vh5

Created

Created container: ensemble-graph-ed576

kserve-ci-e2e-test

InferenceGraphController

ensemble-graph-ed576

InferenceGraphReady

InferenceGraph [ensemble-graph-ed576] is Ready

kserve-ci-e2e-test

deployment-controller

ensemble-graph-ed576

ScalingReplicaSet

Scaled up replica set ensemble-graph-ed576-6d6fdb76b5 from 0 to 1

kserve-ci-e2e-test

replicaset-controller

ensemble-graph-ed576-6d6fdb76b5

SuccessfulCreate

Created pod: ensemble-graph-ed576-6d6fdb76b5-86vh5
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-ed576-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-ed576-predictor

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-ed576-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-ed576-predictor

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x6)

kserve-ci-e2e-test

kubelet

sequence-graph-5e299-6f5777484f-5l654

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503
(x6)

kserve-ci-e2e-test

kubelet

success-200-isvc-54da1-predictor-59846b8c76-rhmxv

Unhealthy

Readiness probe failed: dial tcp 10.133.0.45:8080: connect: connection refused
(x6)

kserve-ci-e2e-test

kubelet

error-404-isvc-54da1-predictor-bf96fd7c-p48j9

Unhealthy

Readiness probe failed: dial tcp 10.133.0.46:8080: connect: connection refused
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

ensemble-graph-ed576

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

ensemble-graph-ed576

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-54da1

InferenceServiceReady

InferenceService [success-200-isvc-54da1] is Ready
(x6)

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-54da1

VirtualServiceCRDNotFound

Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-54da1

InferenceServiceReady

InferenceService [error-404-isvc-54da1] is Ready
(x6)

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-54da1

VirtualServiceCRDNotFound

Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.

kserve-ci-e2e-test

InferenceGraphController

sequence-graph-54da1

UpdateFailed

Failed to update status for InferenceGraph "sequence-graph-54da1": Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "sequence-graph-54da1": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

InferenceGraphController

sequence-graph-54da1

InferenceGraphReady

InferenceGraph [sequence-graph-54da1] is Ready

kserve-ci-e2e-test

deployment-controller

sequence-graph-54da1

ScalingReplicaSet

Scaled up replica set sequence-graph-54da1-594f6b7df8 from 0 to 1

kserve-ci-e2e-test

replicaset-controller

sequence-graph-54da1-594f6b7df8

SuccessfulCreate

Created pod: sequence-graph-54da1-594f6b7df8-f9cxp

kserve-ci-e2e-test

InferenceGraphController

sequence-graph-54da1

InternalError

fails to update InferenceGraph status: Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "sequence-graph-54da1": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

multus

sequence-graph-54da1-594f6b7df8-f9cxp

AddedInterface

Add eth0 [10.133.0.48/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

sequence-graph-54da1-594f6b7df8-f9cxp

Pulled

Container image "quay.io/opendatahub/kserve-router@sha256:dd92bbb6db070213e4821d314fe0d4d7707048c2ffbd06bddadfaae5e74b343a" already present on machine

kserve-ci-e2e-test

kubelet

sequence-graph-54da1-594f6b7df8-f9cxp

Created

Created container: sequence-graph-54da1

kserve-ci-e2e-test

kubelet

sequence-graph-54da1-594f6b7df8-f9cxp

Started

Started container sequence-graph-54da1
(x4)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-54da1-predictor

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x4)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-54da1-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x5)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-54da1-predictor

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x5)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-54da1-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

sequence-graph-54da1

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

sequence-graph-54da1

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kserve-ci-e2e-test

multus

success-200-isvc-3b160-predictor-59b6c8c9d-6ksnz

AddedInterface

Add eth0 [10.133.0.49/23] from ovn-kubernetes

kserve-ci-e2e-test

deployment-controller

error-404-isvc-3b160-predictor

ScalingReplicaSet

Scaled up replica set error-404-isvc-3b160-predictor-788d9b5895 from 0 to 1

kserve-ci-e2e-test

kubelet

error-404-isvc-3b160-predictor-788d9b5895-7hb2d

Created

Created container: kserve-container

kserve-ci-e2e-test

kubelet

error-404-isvc-3b160-predictor-788d9b5895-7hb2d

Started

Started container kserve-container

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-3b160

InternalError

fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-3b160": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-3b160

UpdateFailed

Failed to update status for InferenceService "success-200-isvc-3b160": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-3b160": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

multus

error-404-isvc-3b160-predictor-788d9b5895-7hb2d

AddedInterface

Add eth0 [10.133.0.50/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

success-200-isvc-3b160-predictor-59b6c8c9d-6ksnz

Pulled

Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1426" already present on machine

kserve-ci-e2e-test

replicaset-controller

error-404-isvc-3b160-predictor-788d9b5895

SuccessfulCreate

Created pod: error-404-isvc-3b160-predictor-788d9b5895-7hb2d

kserve-ci-e2e-test

kubelet

error-404-isvc-3b160-predictor-788d9b5895-7hb2d

Pulled

Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1426" already present on machine

kserve-ci-e2e-test

kubelet

ensemble-graph-ed576-6d6fdb76b5-86vh5

Killing

Stopping container ensemble-graph-ed576

kserve-ci-e2e-test

kubelet

error-404-isvc-ed576-predictor-65c7fd45bf-jfsc8

Killing

Stopping container kserve-container

kserve-ci-e2e-test

deployment-controller

success-200-isvc-3b160-predictor

ScalingReplicaSet

Scaled up replica set success-200-isvc-3b160-predictor-59b6c8c9d from 0 to 1

kserve-ci-e2e-test

replicaset-controller

success-200-isvc-3b160-predictor-59b6c8c9d

SuccessfulCreate

Created pod: success-200-isvc-3b160-predictor-59b6c8c9d-6ksnz

kserve-ci-e2e-test

kubelet

success-200-isvc-ed576-predictor-78dc55c9fb-zr27l

Killing

Stopping container kserve-container

kserve-ci-e2e-test

kubelet

success-200-isvc-3b160-predictor-59b6c8c9d-6ksnz

Created

Created container: kserve-container

kserve-ci-e2e-test

kubelet

success-200-isvc-3b160-predictor-59b6c8c9d-6ksnz

Started

Started container kserve-container

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-3b160

InternalError

fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-3b160": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-3b160

UpdateFailed

Failed to update status for InferenceService "error-404-isvc-3b160": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-3b160": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-3b160-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-3b160-predictor

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x6)

kserve-ci-e2e-test

kubelet

ensemble-graph-ed576-6d6fdb76b5-86vh5

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-3b160-predictor

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-3b160-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-d9a0e

UpdateFailed

Failed to update status for InferenceService "error-404-isvc-d9a0e": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-d9a0e": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

replicaset-controller

error-404-isvc-d9a0e-predictor-8b7fd8fb9

SuccessfulCreate

Created pod: error-404-isvc-d9a0e-predictor-8b7fd8fb9-nvzd7

kserve-ci-e2e-test

kubelet

sequence-graph-54da1-594f6b7df8-f9cxp

Killing

Stopping container sequence-graph-54da1

kserve-ci-e2e-test

kubelet

error-404-isvc-54da1-predictor-bf96fd7c-p48j9

Killing

Stopping container kserve-container

kserve-ci-e2e-test

kubelet

success-200-isvc-54da1-predictor-59846b8c76-rhmxv

Killing

Stopping container kserve-container

kserve-ci-e2e-test

multus

error-404-isvc-d9a0e-predictor-8b7fd8fb9-nvzd7

AddedInterface

Add eth0 [10.133.0.52/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

error-404-isvc-d9a0e-predictor-8b7fd8fb9-nvzd7

Pulled

Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1426" already present on machine

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-d9a0e

InternalError

fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-d9a0e": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-d9a0e

UpdateFailed

Failed to update status for InferenceService "success-200-isvc-d9a0e": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-d9a0e": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

deployment-controller

success-200-isvc-d9a0e-predictor

ScalingReplicaSet

Scaled up replica set success-200-isvc-d9a0e-predictor-8587d49fdd from 0 to 1

kserve-ci-e2e-test

replicaset-controller

success-200-isvc-d9a0e-predictor-8587d49fdd

SuccessfulCreate

Created pod: success-200-isvc-d9a0e-predictor-8587d49fdd-wpj2j

kserve-ci-e2e-test

kubelet

success-200-isvc-d9a0e-predictor-8587d49fdd-wpj2j

Started

Started container kserve-container

kserve-ci-e2e-test

kubelet

success-200-isvc-d9a0e-predictor-8587d49fdd-wpj2j

Created

Created container: kserve-container

kserve-ci-e2e-test

kubelet

success-200-isvc-d9a0e-predictor-8587d49fdd-wpj2j

Pulled

Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1426" already present on machine

kserve-ci-e2e-test

multus

success-200-isvc-d9a0e-predictor-8587d49fdd-wpj2j

AddedInterface

Add eth0 [10.133.0.51/23] from ovn-kubernetes

kserve-ci-e2e-test

deployment-controller

error-404-isvc-d9a0e-predictor

ScalingReplicaSet

Scaled up replica set error-404-isvc-d9a0e-predictor-8b7fd8fb9 from 0 to 1

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-d9a0e

InternalError

fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-d9a0e": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

kubelet

error-404-isvc-d9a0e-predictor-8b7fd8fb9-nvzd7

Started

Started container kserve-container

kserve-ci-e2e-test

kubelet

error-404-isvc-d9a0e-predictor-8b7fd8fb9-nvzd7

Created

Created container: kserve-container

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-3b160

InferenceServiceReady

InferenceService [error-404-isvc-3b160] is Ready
(x5)

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-3b160

VirtualServiceCRDNotFound

Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
(x5)

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-3b160

VirtualServiceCRDNotFound

Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-3b160

InferenceServiceReady

InferenceService [success-200-isvc-3b160] is Ready

kserve-ci-e2e-test

deployment-controller

splitter-graph-3b160

ScalingReplicaSet

Scaled up replica set splitter-graph-3b160-555ff4db4b from 0 to 1

kserve-ci-e2e-test

kubelet

splitter-graph-3b160-555ff4db4b-rhktz

FailedMount

MountVolume.SetUp failed for volume "proxy-tls" : secret "splitter-graph-3b160-serving-cert" not found

kserve-ci-e2e-test

InferenceGraphController

splitter-graph-3b160

InternalError

fails to update InferenceGraph status: Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "splitter-graph-3b160": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

InferenceGraphController

splitter-graph-3b160

UpdateFailed

Failed to update status for InferenceGraph "splitter-graph-3b160": Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "splitter-graph-3b160": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

InferenceGraphController

splitter-graph-3b160

InferenceGraphReady

InferenceGraph [splitter-graph-3b160] is Ready

kserve-ci-e2e-test

replicaset-controller

splitter-graph-3b160-555ff4db4b

SuccessfulCreate

Created pod: splitter-graph-3b160-555ff4db4b-rhktz

kserve-ci-e2e-test

kubelet

splitter-graph-3b160-555ff4db4b-rhktz

Started

Started container splitter-graph-3b160

kserve-ci-e2e-test

kubelet

splitter-graph-3b160-555ff4db4b-rhktz

Created

Created container: splitter-graph-3b160

kserve-ci-e2e-test

kubelet

splitter-graph-3b160-555ff4db4b-rhktz

Pulled

Container image "quay.io/opendatahub/kserve-router@sha256:dd92bbb6db070213e4821d314fe0d4d7707048c2ffbd06bddadfaae5e74b343a" already present on machine

kserve-ci-e2e-test

multus

splitter-graph-3b160-555ff4db4b-rhktz

AddedInterface

Add eth0 [10.133.0.53/23] from ovn-kubernetes
(x4)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-3b160-predictor

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-3b160-predictor

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-3b160-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x4)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-3b160-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x6)

kserve-ci-e2e-test

kubelet

sequence-graph-54da1-594f6b7df8-f9cxp

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503

kserve-ci-e2e-test

kubelet

success-200-isvc-3b160-predictor-59b6c8c9d-6ksnz

Killing

Stopping container kserve-container

kserve-ci-e2e-test

multus

success-200-isvc-c5b1b-predictor-5854f8b788-s9zzw

AddedInterface

Add eth0 [10.133.0.54/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

splitter-graph-3b160-555ff4db4b-rhktz

Killing

Stopping container splitter-graph-3b160

kserve-ci-e2e-test

replicaset-controller

success-200-isvc-c5b1b-predictor-5854f8b788

SuccessfulCreate

Created pod: success-200-isvc-c5b1b-predictor-5854f8b788-s9zzw

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-c5b1b

InternalError

fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-c5b1b": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

deployment-controller

success-200-isvc-c5b1b-predictor

ScalingReplicaSet

Scaled up replica set success-200-isvc-c5b1b-predictor-5854f8b788 from 0 to 1

kserve-ci-e2e-test

deployment-controller

error-404-isvc-c5b1b-predictor

ScalingReplicaSet

Scaled up replica set error-404-isvc-c5b1b-predictor-78c899d8d5 from 0 to 1

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-c5b1b

UpdateFailed

Failed to update status for InferenceService "success-200-isvc-c5b1b": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-c5b1b": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-c5b1b

InternalError

fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-c5b1b": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-c5b1b

UpdateFailed

Failed to update status for InferenceService "error-404-isvc-c5b1b": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "error-404-isvc-c5b1b": the object has been modified; please apply your changes to the latest version and try again

kserve-ci-e2e-test

kubelet

error-404-isvc-3b160-predictor-788d9b5895-7hb2d

Killing

Stopping container kserve-container

kserve-ci-e2e-test

replicaset-controller

error-404-isvc-c5b1b-predictor-78c899d8d5

SuccessfulCreate

Created pod: error-404-isvc-c5b1b-predictor-78c899d8d5-r45k2

kserve-ci-e2e-test

kubelet

error-404-isvc-c5b1b-predictor-78c899d8d5-r45k2

Created

Created container: kserve-container

kserve-ci-e2e-test

kubelet

success-200-isvc-c5b1b-predictor-5854f8b788-s9zzw

Started

Started container kserve-container

kserve-ci-e2e-test

kubelet

error-404-isvc-c5b1b-predictor-78c899d8d5-r45k2

Pulled

Container image "quay.io/opendatahub/error-404-isvc:odh-pr-1426" already present on machine
(x7)

kserve-ci-e2e-test

kubelet

success-200-isvc-3b160-predictor-59b6c8c9d-6ksnz

Unhealthy

Readiness probe failed: dial tcp 10.133.0.49:8080: connect: connection refused

kserve-ci-e2e-test

kubelet

success-200-isvc-c5b1b-predictor-5854f8b788-s9zzw

Pulled

Container image "quay.io/opendatahub/success-200-isvc:odh-pr-1426" already present on machine
(x7)

kserve-ci-e2e-test

kubelet

error-404-isvc-3b160-predictor-788d9b5895-7hb2d

Unhealthy

Readiness probe failed: dial tcp 10.133.0.50:8080: connect: connection refused

kserve-ci-e2e-test

kubelet

success-200-isvc-c5b1b-predictor-5854f8b788-s9zzw

Created

Created container: kserve-container

kserve-ci-e2e-test

multus

error-404-isvc-c5b1b-predictor-78c899d8d5-r45k2

AddedInterface

Add eth0 [10.133.0.55/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

error-404-isvc-c5b1b-predictor-78c899d8d5-r45k2

Started

Started container kserve-container
(x6)

kserve-ci-e2e-test

kubelet

error-404-isvc-d9a0e-predictor-8b7fd8fb9-nvzd7

Unhealthy

Readiness probe failed: dial tcp 10.133.0.52:8080: connect: connection refused

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-d9a0e

InferenceServiceReady

InferenceService [success-200-isvc-d9a0e] is Ready
(x5)

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-d9a0e

VirtualServiceCRDNotFound

Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-d9a0e

InferenceServiceReady

InferenceService [error-404-isvc-d9a0e] is Ready
(x6)

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-d9a0e

VirtualServiceCRDNotFound

Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.
(x6)

kserve-ci-e2e-test

kubelet

splitter-graph-3b160-555ff4db4b-rhktz

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503

kserve-ci-e2e-test

deployment-controller

switch-graph-d9a0e

ScalingReplicaSet

Scaled up replica set switch-graph-d9a0e-5c6d76d677 from 0 to 1

kserve-ci-e2e-test

replicaset-controller

switch-graph-d9a0e-5c6d76d677

SuccessfulCreate

Created pod: switch-graph-d9a0e-5c6d76d677-vlrj4

kserve-ci-e2e-test

InferenceGraphController

switch-graph-d9a0e

InferenceGraphReady

InferenceGraph [switch-graph-d9a0e] is Ready

kserve-ci-e2e-test

kubelet

switch-graph-d9a0e-5c6d76d677-vlrj4

Pulled

Container image "quay.io/opendatahub/kserve-router@sha256:dd92bbb6db070213e4821d314fe0d4d7707048c2ffbd06bddadfaae5e74b343a" already present on machine

kserve-ci-e2e-test

kubelet

switch-graph-d9a0e-5c6d76d677-vlrj4

Created

Created container: switch-graph-d9a0e

kserve-ci-e2e-test

kubelet

switch-graph-d9a0e-5c6d76d677-vlrj4

Started

Started container switch-graph-d9a0e

kserve-ci-e2e-test

multus

switch-graph-d9a0e-5c6d76d677-vlrj4

AddedInterface

Add eth0 [10.133.0.56/23] from ovn-kubernetes
(x6)

kserve-ci-e2e-test

kubelet

success-200-isvc-c5b1b-predictor-5854f8b788-s9zzw

Unhealthy

Readiness probe failed: dial tcp 10.133.0.54:8080: connect: connection refused
(x5)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-d9a0e-predictor

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x6)

kserve-ci-e2e-test

kubelet

error-404-isvc-c5b1b-predictor-78c899d8d5-r45k2

Unhealthy

Readiness probe failed: dial tcp 10.133.0.55:8080: connect: connection refused
(x5)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-d9a0e-predictor

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x5)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-d9a0e-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x5)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-d9a0e-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-c5b1b-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-c5b1b-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-c5b1b-predictor

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-c5b1b-predictor

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x5)

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-c5b1b

VirtualServiceCRDNotFound

Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.

kserve-ci-e2e-test

v1beta1Controllers

success-200-isvc-c5b1b

InferenceServiceReady

InferenceService [success-200-isvc-c5b1b] is Ready
(x5)

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-c5b1b

VirtualServiceCRDNotFound

Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true.

kserve-ci-e2e-test

horizontal-pod-autoscaler

switch-graph-d9a0e

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kserve-ci-e2e-test

horizontal-pod-autoscaler

switch-graph-d9a0e

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kserve-ci-e2e-test

v1beta1Controllers

error-404-isvc-c5b1b

InferenceServiceReady

InferenceService [error-404-isvc-c5b1b] is Ready

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-d9a0e-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-d9a0e-predictor

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

kserve-ci-e2e-test

horizontal-pod-autoscaler

switch-graph-d9a0e

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

kserve-ci-e2e-test

horizontal-pod-autoscaler

switch-graph-d9a0e

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)

kserve-ci-e2e-test

replicaset-controller

splitter-graph-c5b1b-59bd77c885

SuccessfulCreate

Created pod: splitter-graph-c5b1b-59bd77c885-m9xp8

kserve-ci-e2e-test

kubelet

splitter-graph-c5b1b-59bd77c885-m9xp8

FailedMount

MountVolume.SetUp failed for volume "proxy-tls" : secret "splitter-graph-c5b1b-serving-cert" not found

kserve-ci-e2e-test

InferenceGraphController

splitter-graph-c5b1b

InferenceGraphReady

InferenceGraph [splitter-graph-c5b1b] is Ready

kserve-ci-e2e-test

deployment-controller

splitter-graph-c5b1b

ScalingReplicaSet

Scaled up replica set splitter-graph-c5b1b-59bd77c885 from 0 to 1

kserve-ci-e2e-test

kubelet

splitter-graph-c5b1b-59bd77c885-m9xp8

Created

Created container: splitter-graph-c5b1b

kserve-ci-e2e-test

kubelet

splitter-graph-c5b1b-59bd77c885-m9xp8

Pulled

Container image "quay.io/opendatahub/kserve-router@sha256:dd92bbb6db070213e4821d314fe0d4d7707048c2ffbd06bddadfaae5e74b343a" already present on machine

kserve-ci-e2e-test

multus

splitter-graph-c5b1b-59bd77c885-m9xp8

AddedInterface

Add eth0 [10.133.0.57/23] from ovn-kubernetes

kserve-ci-e2e-test

kubelet

splitter-graph-c5b1b-59bd77c885-m9xp8

Started

Started container splitter-graph-c5b1b
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-c5b1b-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

error-404-isvc-c5b1b-predictor

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-c5b1b-predictor

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x3)

kserve-ci-e2e-test

horizontal-pod-autoscaler

success-200-isvc-c5b1b-predictor

FailedGetResourceMetric

failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

splitter-graph-c5b1b

FailedGetResourceMetric

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
(x2)

kserve-ci-e2e-test

horizontal-pod-autoscaler

splitter-graph-c5b1b

FailedComputeMetricsReplicas

invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

kserve-ci-e2e-test

kubelet

splitter-graph-c5b1b-59bd77c885-m9xp8

Killing

Stopping container splitter-graph-c5b1b

kserve-ci-e2e-test

kubelet

error-404-isvc-c5b1b-predictor-78c899d8d5-r45k2

Killing

Stopping container kserve-container

kserve-ci-e2e-test

kubelet

success-200-isvc-c5b1b-predictor-5854f8b788-s9zzw

Killing

Stopping container kserve-container
(x6)

kserve-ci-e2e-test

kubelet

splitter-graph-c5b1b-59bd77c885-m9xp8

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503

kserve-ci-e2e-test

kubelet

success-200-isvc-d9a0e-predictor-8587d49fdd-wpj2j

Killing

Stopping container kserve-container

kserve-ci-e2e-test

kubelet

switch-graph-d9a0e-5c6d76d677-vlrj4

Killing

Stopping container switch-graph-d9a0e

kserve-ci-e2e-test

kubelet

error-404-isvc-d9a0e-predictor-8b7fd8fb9-nvzd7

Killing

Stopping container kserve-container
(x7)

kserve-ci-e2e-test

kubelet

success-200-isvc-d9a0e-predictor-8587d49fdd-wpj2j

Unhealthy

Readiness probe failed: dial tcp 10.133.0.51:8080: connect: connection refused
(x5)

kserve-ci-e2e-test

kubelet

switch-graph-d9a0e-5c6d76d677-vlrj4

Unhealthy

Readiness probe failed: HTTP probe failed with statuscode: 503