| Time | Namespace | Component | RelatedObject | Reason | Message |
|---|---|---|---|---|---|
kserve-ci-e2e-test |
model-chainer-raw-4bd9d-b569d849-26rp9 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-4bd9d-b569d849-26rp9 to ip-10-0-133-158.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-61a06-predictor-cf9788b8b-9nls5 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-61a06-predictor-cf9788b8b-9nls5 to ip-10-0-136-162.ec2.internal | ||
kserve-ci-e2e-test |
isvc-secondary-daad18-predictor-58999cdf79-vzkg4 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-secondary-daad18-predictor-58999cdf79-vzkg4 to ip-10-0-136-162.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-hpa-3991c-predictor-65d954f97f-xkfv4 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3991c-predictor-65d954f97f-xkfv4 to ip-10-0-136-162.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-61a06-predictor-58dddf4bf8-8wlzm |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-61a06-predictor-58dddf4bf8-8wlzm to ip-10-0-136-162.ec2.internal | ||
kserve-ci-e2e-test |
isvc-primary-daad18-predictor-7d747cb478-h4pvv |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-primary-daad18-predictor-7d747cb478-h4pvv to ip-10-0-136-162.ec2.internal | ||
kserve-ci-e2e-test |
isvc-raw-sklearn-batcher-03d5d-predictor-6c5d487c5c-67tvj |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-raw-sklearn-batcher-03d5d-predictor-6c5d487c5c-67tvj to ip-10-0-136-162.ec2.internal | ||
kserve-ci-e2e-test |
isvc-init-fail-e57b3c-predictor-845664f98f-bsl29 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-init-fail-e57b3c-predictor-845664f98f-bsl29 to ip-10-0-136-162.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-scale-raw-1f208-predictor-55ff5c688f-d24z8 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-scale-raw-1f208-predictor-55ff5c688f-d24z8 to ip-10-0-136-162.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-4bd9d-predictor-cb5d76c5-xrhlh |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-4bd9d-predictor-cb5d76c5-xrhlh to ip-10-0-136-162.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-e5dc9-predictor-b76fc85f7-5tnlr |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-e5dc9-predictor-b76fc85f7-5tnlr to ip-10-0-136-162.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-hpa-3991c-predictor-6d58c8fd99-2n4qc |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3991c-predictor-6d58c8fd99-2n4qc to ip-10-0-136-162.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-hpa-3991c-bbfb6647c-5twzp |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-hpa-3991c-bbfb6647c-5twzp to ip-10-0-133-158.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-runtime-06bac-predictor-6f4db74cd7-hc5h8 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-runtime-06bac-predictor-6f4db74cd7-hc5h8 to ip-10-0-136-162.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-4bd9d-predictor-85f57b9f67-q49pb |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-4bd9d-predictor-85f57b9f67-q49pb to ip-10-0-136-162.ec2.internal | ||
kserve-ci-e2e-test |
deployment-controller |
isvc-raw-sklearn-batcher-03d5d-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-raw-sklearn-batcher-03d5d-predictor-6c5d487c5c from 0 to 1 | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-03d5d |
UpdateFailed |
Failed to update status for InferenceService "isvc-raw-sklearn-batcher-03d5d": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-03d5d": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-03d5d |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-03d5d": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
replicaset-controller |
isvc-raw-sklearn-batcher-03d5d-predictor-6c5d487c5c |
SuccessfulCreate |
Created pod: isvc-raw-sklearn-batcher-03d5d-predictor-6c5d487c5c-67tvj | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-03d5d-predictor-6c5d487c5c-67tvj |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:6917b7549dba28168fe9ddab83ad49bb5876a127ac47aab2659411acc81a41e6" | |
kserve-ci-e2e-test |
multus |
isvc-raw-sklearn-batcher-03d5d-predictor-6c5d487c5c-67tvj |
AddedInterface |
Add eth0 [10.132.0.37/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-03d5d-predictor-6c5d487c5c-67tvj |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-03d5d-predictor-6c5d487c5c-67tvj |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:6917b7549dba28168fe9ddab83ad49bb5876a127ac47aab2659411acc81a41e6" in 4.144s (4.144s including waiting). Image size: 301280147 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-03d5d-predictor-6c5d487c5c-67tvj |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-03d5d-predictor-6c5d487c5c-67tvj |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1428" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-03d5d-predictor-6c5d487c5c-67tvj |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1428" in 13.409s (13.409s including waiting). Image size: 1560596387 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-03d5d-predictor-6c5d487c5c-67tvj |
Pulling |
Pulling image "quay.io/opendatahub/kserve-agent@sha256:56140db5c247b351ac2dad1a46e24a34c7c91836ee8418891bd73409e3a23542" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-03d5d-predictor-6c5d487c5c-67tvj |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-03d5d-predictor-6c5d487c5c-67tvj |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-03d5d-predictor-6c5d487c5c-67tvj |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-03d5d-predictor-6c5d487c5c-67tvj |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-03d5d-predictor-6c5d487c5c-67tvj |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:56140db5c247b351ac2dad1a46e24a34c7c91836ee8418891bd73409e3a23542" in 2.15s (2.15s including waiting). Image size: 237897782 bytes. | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-03d5d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-03d5d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-03d5d |
InferenceServiceReady |
InferenceService [isvc-raw-sklearn-batcher-03d5d] is Ready | |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-03d5d |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-4bd9d |
UpdateFailed |
Failed to update status for InferenceService "isvc-xgboost-graph-raw-4bd9d": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-graph-raw-4bd9d": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-4bd9d-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-4bd9d-predictor-cb5d76c5 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-03d5d-predictor-6c5d487c5c-67tvj |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-4bd9d-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-4bd9d-predictor-85f57b9f67 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-4bd9d |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-4bd9d": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-4bd9d": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-4bd9d-predictor-cb5d76c5 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-4bd9d-predictor-cb5d76c5-xrhlh | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-03d5d-predictor-6c5d487c5c-67tvj |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-4bd9d-predictor-85f57b9f67 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-4bd9d-predictor-85f57b9f67-q49pb | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-03d5d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-03d5d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-4bd9d-predictor-85f57b9f67-q49pb |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-4bd9d-predictor-85f57b9f67-q49pb |
AddedInterface |
Add eth0 [10.132.0.38/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-4bd9d-predictor-cb5d76c5-xrhlh |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-4bd9d-predictor-cb5d76c5-xrhlh |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-4bd9d-predictor-cb5d76c5-xrhlh |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6917b7549dba28168fe9ddab83ad49bb5876a127ac47aab2659411acc81a41e6" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-4bd9d-predictor-cb5d76c5-xrhlh |
AddedInterface |
Add eth0 [10.132.0.39/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-4bd9d-predictor-85f57b9f67-q49pb |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6917b7549dba28168fe9ddab83ad49bb5876a127ac47aab2659411acc81a41e6" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-4bd9d-predictor-85f57b9f67-q49pb |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-4bd9d-predictor-cb5d76c5-xrhlh |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-4bd9d-predictor-85f57b9f67-q49pb |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1428" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-4bd9d-predictor-85f57b9f67-q49pb |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-4bd9d-predictor-85f57b9f67-q49pb |
Started |
Started container kserve-container | |
| (x12) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-03d5d-predictor-6c5d487c5c-67tvj |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.37:8080: connect: connection refused |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-03d5d-predictor-6c5d487c5c-67tvj |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-4bd9d-predictor-cb5d76c5-xrhlh |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-4bd9d-predictor-cb5d76c5-xrhlh |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-4bd9d-predictor-cb5d76c5-xrhlh |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 18.208s (18.208s including waiting). Image size: 1306414327 bytes. | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-4bd9d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-4bd9d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-4bd9d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-4bd9d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-4bd9d-predictor-85f57b9f67-q49pb |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.38:8080: connect: connection refused |
| (x7) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-4bd9d-predictor-cb5d76c5-xrhlh |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.39:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-4bd9d |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-4bd9d |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-4bd9d] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-4bd9d |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-4bd9d |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-4bd9d] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-4bd9d |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-4bd9d-b569d849 from 0 to 1 | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-4bd9d |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-4bd9d] is Ready | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-4bd9d-b569d849 |
SuccessfulCreate |
Created pod: model-chainer-raw-4bd9d-b569d849-26rp9 | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-4bd9d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-4bd9d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-4bd9d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-4bd9d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
multus |
model-chainer-raw-4bd9d-b569d849-26rp9 |
AddedInterface |
Add eth0 [10.134.0.21/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-4bd9d-b569d849-26rp9 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-router@sha256:85893624ca442ac717d0ce5a9411858686c59f207eb581069a52d014f60caab4" | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-4bd9d-b569d849-26rp9 |
Created |
Created container: model-chainer-raw-4bd9d | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-4bd9d-b569d849-26rp9 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:85893624ca442ac717d0ce5a9411858686c59f207eb581069a52d014f60caab4" in 2.381s (2.381s including waiting). Image size: 216233529 bytes. | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-4bd9d-b569d849-26rp9 |
Started |
Started container model-chainer-raw-4bd9d | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-4bd9d-b569d849-26rp9 |
Killing |
Stopping container model-chainer-raw-4bd9d | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-hpa-3991c-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-hpa-3991c-predictor-65d954f97f from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-hpa-3991c-predictor-65d954f97f |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-hpa-3991c-predictor-65d954f97f-xkfv4 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-hpa-3991c-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-hpa-3991c-predictor-6d58c8fd99 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-hpa-3991c-predictor-6d58c8fd99 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-hpa-3991c-predictor-6d58c8fd99-2n4qc | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-4bd9d-predictor-85f57b9f67-q49pb |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-4bd9d-predictor-cb5d76c5-xrhlh |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3991c-predictor-6d58c8fd99-2n4qc |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-hpa-3991c-predictor-6d58c8fd99-2n4qc |
AddedInterface |
Add eth0 [10.132.0.41/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3991c-predictor-65d954f97f-xkfv4 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6917b7549dba28168fe9ddab83ad49bb5876a127ac47aab2659411acc81a41e6" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-hpa-3991c-predictor-65d954f97f-xkfv4 |
AddedInterface |
Add eth0 [10.132.0.40/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3991c-predictor-65d954f97f-xkfv4 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3991c-predictor-6d58c8fd99-2n4qc |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3991c-predictor-65d954f97f-xkfv4 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3991c-predictor-6d58c8fd99-2n4qc |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6917b7549dba28168fe9ddab83ad49bb5876a127ac47aab2659411acc81a41e6" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3991c-predictor-65d954f97f-xkfv4 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3991c-predictor-6d58c8fd99-2n4qc |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3991c-predictor-6d58c8fd99-2n4qc |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3991c-predictor-6d58c8fd99-2n4qc |
Pulled |
Container image "kserve/xgbserver:latest" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3991c-predictor-65d954f97f-xkfv4 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3991c-predictor-65d954f97f-xkfv4 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1428" already present on machine | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-3991c-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-3991c-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-4bd9d-b569d849-26rp9 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-3991c-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-3991c-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x7) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3991c-predictor-6d58c8fd99-2n4qc |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.41:8080: connect: connection refused |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-3991c |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-3991c |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-hpa-3991c] is Ready | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3991c-predictor-65d954f97f-xkfv4 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.40:8080: connect: connection refused |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-3991c |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-3991c |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-hpa-3991c] is Ready | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-3991c-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-3991c-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-3991c |
InternalError |
fails to update InferenceGraph status: Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "model-chainer-raw-hpa-3991c": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-3991c |
UpdateFailed |
Failed to update status for InferenceGraph "model-chainer-raw-hpa-3991c": Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "model-chainer-raw-hpa-3991c": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-3991c |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-hpa-3991c] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-hpa-3991c |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-hpa-3991c-bbfb6647c from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-hpa-3991c-bbfb6647c |
SuccessfulCreate |
Created pod: model-chainer-raw-hpa-3991c-bbfb6647c-5twzp | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-3991c-bbfb6647c-5twzp |
Created |
Created container: model-chainer-raw-hpa-3991c | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-3991c-bbfb6647c-5twzp |
Started |
Started container model-chainer-raw-hpa-3991c | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-3991c-bbfb6647c-5twzp |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:85893624ca442ac717d0ce5a9411858686c59f207eb581069a52d014f60caab4" already present on machine | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-hpa-3991c-bbfb6647c-5twzp |
AddedInterface |
Add eth0 [10.134.0.22/23] from ovn-kubernetes | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-3991c-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-3991c-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-3991c-bbfb6647c-5twzp |
Killing |
Stopping container model-chainer-raw-hpa-3991c | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3991c-predictor-65d954f97f-xkfv4 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-61a06-predictor-cf9788b8b |
SuccessfulCreate |
Created pod: message-dumper-raw-61a06-predictor-cf9788b8b-9nls5 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3991c-predictor-6d58c8fd99-2n4qc |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-61a06-predictor-cf9788b8b-9nls5 |
AddedInterface |
Add eth0 [10.132.0.42/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-61a06-predictor-cf9788b8b-9nls5 |
Pulling |
Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-61a06 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-61a06": the object has been modified; please apply your changes to the latest version and try again | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-61a06 |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-61a06": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-61a06": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-61a06-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-61a06-predictor-cf9788b8b from 0 to 1 | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-61a06 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-61a06-predictor-cf9788b8b-9nls5 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-61a06-predictor-cf9788b8b-9nls5 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-61a06-predictor-cf9788b8b-9nls5 |
Pulled |
Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.116s (1.116s including waiting). Image size: 14813193 bytes. | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-61a06 |
InferenceServiceReady |
InferenceService [message-dumper-raw-61a06] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-61a06-predictor-58dddf4bf8-8wlzm |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6917b7549dba28168fe9ddab83ad49bb5876a127ac47aab2659411acc81a41e6" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-61a06-predictor-58dddf4bf8-8wlzm |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-61a06-predictor-58dddf4bf8-8wlzm |
AddedInterface |
Add eth0 [10.132.0.43/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-61a06 |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-61a06": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-61a06": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-61a06-predictor-58dddf4bf8-8wlzm |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-61a06-predictor-58dddf4bf8 |
SuccessfulCreate |
Created pod: isvc-logger-raw-61a06-predictor-58dddf4bf8-8wlzm | |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-61a06-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-61a06-predictor-58dddf4bf8 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-61a06-predictor-58dddf4bf8-8wlzm |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-61a06-predictor-58dddf4bf8-8wlzm |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1428" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-61a06-predictor-58dddf4bf8-8wlzm |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:56140db5c247b351ac2dad1a46e24a34c7c91836ee8418891bd73409e3a23542" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-61a06-predictor-58dddf4bf8-8wlzm |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-61a06-predictor-58dddf4bf8-8wlzm |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-61a06-predictor-58dddf4bf8-8wlzm |
Started |
Started container agent | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-61a06-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-61a06-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-61a06-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-61a06-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-3991c-bbfb6647c-5twzp |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-61a06 |
InferenceServiceReady |
InferenceService [isvc-logger-raw-61a06] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-61a06 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-61a06-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-61a06-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-61a06-predictor-cf9788b8b-9nls5 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-scale-raw-1f208-predictor-55ff5c688f |
SuccessfulCreate |
Created pod: isvc-sklearn-scale-raw-1f208-predictor-55ff5c688f-d24z8 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-scale-raw-1f208-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-scale-raw-1f208-predictor-55ff5c688f from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-61a06-predictor-58dddf4bf8-8wlzm |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-1f208 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-scale-raw-1f208": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scale-raw-1f208": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-61a06-predictor-58dddf4bf8-8wlzm |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-1f208-predictor-55ff5c688f-d24z8 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-1f208-predictor-55ff5c688f-d24z8 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6917b7549dba28168fe9ddab83ad49bb5876a127ac47aab2659411acc81a41e6" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-1f208-predictor-55ff5c688f-d24z8 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-scale-raw-1f208-predictor-55ff5c688f-d24z8 |
AddedInterface |
Add eth0 [10.132.0.44/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-1f208-predictor-55ff5c688f-d24z8 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1428" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-1f208-predictor-55ff5c688f-d24z8 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-1f208-predictor-55ff5c688f-d24z8 |
Started |
Started container kserve-container | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-61a06-predictor-58dddf4bf8-8wlzm |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-61a06-predictor-58dddf4bf8-8wlzm |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.43:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-1f208-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-1f208-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x8) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-1f208-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x8) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-1f208-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-1f208 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-1f208 |
InferenceServiceReady |
InferenceService [isvc-sklearn-scale-raw-1f208] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-1f208-predictor-55ff5c688f-d24z8 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-daad18 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-daad18": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-daad18 |
UpdateFailed |
Failed to update status for InferenceService "isvc-primary-daad18": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-daad18": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-primary-daad18-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-primary-daad18-predictor-7d747cb478 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-primary-daad18-predictor-7d747cb478 |
SuccessfulCreate |
Created pod: isvc-primary-daad18-predictor-7d747cb478-h4pvv | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-daad18-predictor-7d747cb478-h4pvv |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-daad18-predictor-7d747cb478-h4pvv |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-primary-daad18-predictor-7d747cb478-h4pvv |
AddedInterface |
Add eth0 [10.132.0.45/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-daad18-predictor-7d747cb478-h4pvv |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6917b7549dba28168fe9ddab83ad49bb5876a127ac47aab2659411acc81a41e6" already present on machine | |
| (x17) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-1f208-predictor-55ff5c688f-d24z8 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.44:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-primary-daad18-predictor-7d747cb478-h4pvv |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1428" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-daad18-predictor-7d747cb478-h4pvv |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-daad18-predictor-7d747cb478-h4pvv |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-daad18-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-daad18-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-primary-daad18-predictor-7d747cb478-h4pvv |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.45:8080: connect: connection refused |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-daad18 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-daad18 |
InferenceServiceReady |
InferenceService [isvc-primary-daad18] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
isvc-secondary-daad18-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-secondary-daad18-predictor-58999cdf79 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-daad18 |
UpdateFailed |
Failed to update status for InferenceService "isvc-secondary-daad18": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-daad18": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-secondary-daad18-predictor-58999cdf79 |
SuccessfulCreate |
Created pod: isvc-secondary-daad18-predictor-58999cdf79-vzkg4 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-daad18 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-daad18": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-secondary-daad18-predictor-58999cdf79-vzkg4 |
AddedInterface |
Add eth0 [10.132.0.46/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-daad18-predictor-58999cdf79-vzkg4 |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-daad18-predictor-58999cdf79-vzkg4 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6917b7549dba28168fe9ddab83ad49bb5876a127ac47aab2659411acc81a41e6" already present on machine |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-daad18-predictor-58999cdf79-vzkg4 |
Created |
Created container: storage-initializer |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-daad18-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x14) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-daad18 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-daad18-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-daad18-predictor-58999cdf79-vzkg4 |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-secondary-daad18-predictor-58999cdf79-vzkg4_kserve-ci-e2e-test(f334e01d-09cc-425c-8ef3-0d3cbff4f175) | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-daad18-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-daad18-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
multus |
isvc-init-fail-e57b3c-predictor-845664f98f-bsl29 |
AddedInterface |
Add eth0 [10.132.0.47/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-e57b3c |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-e57b3c": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-primary-daad18-predictor-7d747cb478-h4pvv |
Killing |
Stopping container kserve-container | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-e57b3c |
UpdateFailed |
Failed to update status for InferenceService "isvc-init-fail-e57b3c": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-e57b3c": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
replicaset-controller |
isvc-init-fail-e57b3c-predictor-845664f98f |
SuccessfulCreate |
Created pod: isvc-init-fail-e57b3c-predictor-845664f98f-bsl29 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-init-fail-e57b3c-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-init-fail-e57b3c-predictor-845664f98f from 0 to 1 | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-e57b3c-predictor-845664f98f-bsl29 |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-e57b3c-predictor-845664f98f-bsl29 |
Started |
Started container storage-initializer |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-e57b3c |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-e57b3c-predictor-845664f98f-bsl29 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6917b7549dba28168fe9ddab83ad49bb5876a127ac47aab2659411acc81a41e6" already present on machine |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-e5dc9-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-e5dc9-predictor-b76fc85f7 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-e57b3c-predictor-845664f98f-bsl29 |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-init-fail-e57b3c-predictor-845664f98f-bsl29_kserve-ci-e2e-test(48d460f5-eb59-4977-a544-3c3a8107b8d0) | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-e5dc9-predictor-b76fc85f7-5tnlr |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-e5dc9-predictor-b76fc85f7-5tnlr |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-e5dc9-predictor-b76fc85f7-5tnlr |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6917b7549dba28168fe9ddab83ad49bb5876a127ac47aab2659411acc81a41e6" already present on machine | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-e5dc9 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-e5dc9": the object has been modified; please apply your changes to the latest version and try again | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-e5dc9 |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-e5dc9": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-e5dc9": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
multus |
raw-sklearn-e5dc9-predictor-b76fc85f7-5tnlr |
AddedInterface |
Add eth0 [10.132.0.48/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-e5dc9-predictor-b76fc85f7 |
SuccessfulCreate |
Created pod: raw-sklearn-e5dc9-predictor-b76fc85f7-5tnlr | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-e5dc9-predictor-b76fc85f7-5tnlr |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1428" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-e5dc9-predictor-b76fc85f7-5tnlr |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-e5dc9-predictor-b76fc85f7-5tnlr |
Created |
Created container: kserve-container | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-e5dc9-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-e5dc9-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x8) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-e5dc9-predictor-b76fc85f7-5tnlr |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.48:8080: connect: connection refused |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-e5dc9-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-e5dc9-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x14) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-e5dc9 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-e5dc9 |
InferenceServiceReady |
InferenceService [raw-sklearn-e5dc9] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-runtime-06bac-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-runtime-06bac-predictor-6f4db74cd7 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-e5dc9-predictor-b76fc85f7-5tnlr |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-06bac |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-runtime-06bac": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-06bac": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-runtime-06bac-predictor-6f4db74cd7 |
SuccessfulCreate |
Created pod: raw-sklearn-runtime-06bac-predictor-6f4db74cd7-hc5h8 | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-06bac-predictor-6f4db74cd7-hc5h8 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6917b7549dba28168fe9ddab83ad49bb5876a127ac47aab2659411acc81a41e6" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-06bac-predictor-6f4db74cd7-hc5h8 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-06bac-predictor-6f4db74cd7-hc5h8 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
raw-sklearn-runtime-06bac-predictor-6f4db74cd7-hc5h8 |
AddedInterface |
Add eth0 [10.132.0.49/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-06bac-predictor-6f4db74cd7-hc5h8 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1428" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-06bac-predictor-6f4db74cd7-hc5h8 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-06bac-predictor-6f4db74cd7-hc5h8 |
Started |
Started container kserve-container | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-06bac-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-06bac-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x8) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-06bac-predictor-6f4db74cd7-hc5h8 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.49:8080: connect: connection refused |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-06bac-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-06bac-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-06bac |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-06bac |
InferenceServiceReady |
InferenceService [raw-sklearn-runtime-06bac] is Ready | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-06bac-predictor-6f4db74cd7-hc5h8 |
Killing |
Stopping container kserve-container |