| Time | Namespace | Component | RelatedObject | Reason | Message |
|---|---|---|---|---|---|
kserve-ci-e2e-test |
isvc-init-fail-b43ebd-predictor-84bd96d595-89tkv |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-init-fail-b43ebd-predictor-84bd96d595-89tkv to ip-10-0-135-144.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-scale-raw-954a2-predictor-877cc765-jf86g |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-scale-raw-954a2-predictor-877cc765-jf86g to ip-10-0-135-144.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-5512a-predictor-79fc6d66bc-djd6g |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-5512a-predictor-79fc6d66bc-djd6g to ip-10-0-135-144.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-hpa-724bc-d588cbf89-2tdkd |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-hpa-724bc-d588cbf89-2tdkd to ip-10-0-129-182.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-3b643-predictor-584c5864d8-42v2g |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b643-predictor-584c5864d8-42v2g to ip-10-0-135-144.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-hpa-724bc-predictor-8485c796b7-nglk8 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-724bc-predictor-8485c796b7-nglk8 to ip-10-0-135-144.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-runtime-82bab-predictor-c74656cbf-cdn6c |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-runtime-82bab-predictor-c74656cbf-cdn6c to ip-10-0-135-144.ec2.internal | ||
kserve-ci-e2e-test |
isvc-secondary-89be61-predictor-5bdc697fb7-9ndcp |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-secondary-89be61-predictor-5bdc697fb7-9ndcp to ip-10-0-135-144.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-3b643-7f87dff547-9r7xz |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-3b643-7f87dff547-9r7xz to ip-10-0-129-182.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-3b643-predictor-94775489-88l6s |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b643-predictor-94775489-88l6s to ip-10-0-135-144.ec2.internal | ||
kserve-ci-e2e-test |
isvc-primary-89be61-predictor-789bf94c5d-99btm |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-primary-89be61-predictor-789bf94c5d-99btm to ip-10-0-135-144.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-74f70-predictor-86f49fcb48-hxk5b |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-74f70-predictor-86f49fcb48-hxk5b to ip-10-0-135-144.ec2.internal | ||
kserve-ci-e2e-test |
isvc-raw-sklearn-batcher-70fe9-predictor-667cd4c84d-rxgg8 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-raw-sklearn-batcher-70fe9-predictor-667cd4c84d-rxgg8 to ip-10-0-135-144.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-hpa-724bc-predictor-795997bf5d-xwcpq |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-724bc-predictor-795997bf5d-xwcpq to ip-10-0-135-144.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-74f70-predictor-767887b594-gh5xm |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-74f70-predictor-767887b594-gh5xm to ip-10-0-129-182.ec2.internal | ||
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-70fe9 |
UpdateFailed |
Failed to update status for InferenceService "isvc-raw-sklearn-batcher-70fe9": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-70fe9": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-70fe9-predictor-667cd4c84d-rxgg8 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:0784282e0aaaff940f9515f44b229c4f93907c9657748d25ae94f59437bac840" | |
kserve-ci-e2e-test |
multus |
isvc-raw-sklearn-batcher-70fe9-predictor-667cd4c84d-rxgg8 |
AddedInterface |
Add eth0 [10.133.0.22/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-raw-sklearn-batcher-70fe9-predictor-667cd4c84d |
SuccessfulCreate |
Created pod: isvc-raw-sklearn-batcher-70fe9-predictor-667cd4c84d-rxgg8 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-70fe9 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-70fe9": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-raw-sklearn-batcher-70fe9-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-raw-sklearn-batcher-70fe9-predictor-667cd4c84d from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-70fe9-predictor-667cd4c84d-rxgg8 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:0784282e0aaaff940f9515f44b229c4f93907c9657748d25ae94f59437bac840" in 2.991s (2.991s including waiting). Image size: 299659693 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-70fe9-predictor-667cd4c84d-rxgg8 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-70fe9-predictor-667cd4c84d-rxgg8 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-70fe9-predictor-667cd4c84d-rxgg8 |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1326" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-70fe9-predictor-667cd4c84d-rxgg8 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-70fe9-predictor-667cd4c84d-rxgg8 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-70fe9-predictor-667cd4c84d-rxgg8 |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1326" in 13.112s (13.112s including waiting). Image size: 1562961329 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-70fe9-predictor-667cd4c84d-rxgg8 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-agent@sha256:f2fb11757d99ef46b1388e1609ace6b5f516fdebd4ee81c172186c5de9dc18b9" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-70fe9-predictor-667cd4c84d-rxgg8 |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-70fe9-predictor-667cd4c84d-rxgg8 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:f2fb11757d99ef46b1388e1609ace6b5f516fdebd4ee81c172186c5de9dc18b9" in 2.37s (2.37s including waiting). Image size: 237894735 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-70fe9-predictor-667cd4c84d-rxgg8 |
Created |
Created container: agent | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-70fe9-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-70fe9-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-70fe9 |
InferenceServiceReady |
InferenceService [isvc-raw-sklearn-batcher-70fe9] is Ready | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-70fe9 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-3b643 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-3b643": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-3b643": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-70fe9-predictor-667cd4c84d-rxgg8 |
Killing |
Stopping container kserve-container | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-70fe9-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-3b643-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-3b643-predictor-584c5864d8 from 0 to 1 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-3b643-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-3b643-predictor-94775489 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-3b643-predictor-94775489 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-3b643-predictor-94775489-88l6s | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-3b643 |
UpdateFailed |
Failed to update status for InferenceService "isvc-xgboost-graph-raw-3b643": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-graph-raw-3b643": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-3b643-predictor-584c5864d8 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-3b643-predictor-584c5864d8-42v2g | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-70fe9-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-70fe9-predictor-667cd4c84d-rxgg8 |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-3b643-predictor-584c5864d8-42v2g |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-3b643-predictor-584c5864d8-42v2g |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-3b643-predictor-94775489-88l6s |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-3b643-predictor-94775489-88l6s |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-3b643-predictor-94775489-88l6s |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0784282e0aaaff940f9515f44b229c4f93907c9657748d25ae94f59437bac840" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-3b643-predictor-94775489-88l6s |
AddedInterface |
Add eth0 [10.133.0.24/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-3b643-predictor-584c5864d8-42v2g |
AddedInterface |
Add eth0 [10.133.0.23/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-3b643-predictor-584c5864d8-42v2g |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0784282e0aaaff940f9515f44b229c4f93907c9657748d25ae94f59437bac840" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-3b643-predictor-584c5864d8-42v2g |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1326" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-3b643-predictor-584c5864d8-42v2g |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-3b643-predictor-94775489-88l6s |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-3b643-predictor-584c5864d8-42v2g |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-3b643-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-3b643-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x12) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-70fe9-predictor-667cd4c84d-rxgg8 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.22:8080: connect: connection refused |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-70fe9-predictor-667cd4c84d-rxgg8 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-3b643-predictor-94775489-88l6s |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-3b643-predictor-94775489-88l6s |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-3b643-predictor-94775489-88l6s |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 17.074s (17.074s including waiting). Image size: 1306229499 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-70fe9-predictor-667cd4c84d-rxgg8 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.22:8080: i/o timeout | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-70fe9-predictor-667cd4c84d-rxgg8 |
Unhealthy |
Readiness probe failed: Get "http://10.133.0.22:9081/": context deadline exceeded (Client.Timeout exceeded while awaiting headers) | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-3b643-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-3b643-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-3b643-predictor-584c5864d8-42v2g |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.23:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-3b643 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-3b643] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-3b643 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-3b643 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-3b643 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-3b643] is Ready | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-3b643-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-3b643-7f87dff547 |
SuccessfulCreate |
Created pod: model-chainer-raw-3b643-7f87dff547-9r7xz | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-3b643-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-3b643-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-3b643-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-3b643 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-3b643] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-3b643 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-3b643-7f87dff547 from 0 to 1 | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-3b643-7f87dff547-9r7xz |
AddedInterface |
Add eth0 [10.132.0.34/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-3b643-7f87dff547-9r7xz |
Pulling |
Pulling image "quay.io/opendatahub/kserve-router@sha256:743cff1ceb833e46d1b8e65ba261d1d549e2e90054da6d533b54a9a2f13462c7" | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-3b643-7f87dff547-9r7xz |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:743cff1ceb833e46d1b8e65ba261d1d549e2e90054da6d533b54a9a2f13462c7" in 2.622s (2.622s including waiting). Image size: 216230482 bytes. | |
| (x2) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-3b643-7f87dff547-9r7xz |
Created |
Created container: model-chainer-raw-3b643 |
| (x2) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-3b643-7f87dff547-9r7xz |
Started |
Started container model-chainer-raw-3b643 |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-3b643-7f87dff547-9r7xz |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:743cff1ceb833e46d1b8e65ba261d1d549e2e90054da6d533b54a9a2f13462c7" already present on machine | |
| (x3) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-3b643-7f87dff547-9r7xz |
BackOff |
Back-off restarting failed container model-chainer-raw-3b643 in pod model-chainer-raw-3b643-7f87dff547-9r7xz_kserve-ci-e2e-test(2c95eb78-181b-4996-a4a5-b1f30325d319) |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-3b643-predictor-584c5864d8-42v2g |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-724bc |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-hpa-724bc": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-hpa-724bc": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-hpa-724bc-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-hpa-724bc-predictor-795997bf5d from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-724bc-predictor-8485c796b7-nglk8 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0784282e0aaaff940f9515f44b229c4f93907c9657748d25ae94f59437bac840" already present on machine | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-hpa-724bc-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-hpa-724bc-predictor-8485c796b7 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-724bc-predictor-795997bf5d-xwcpq |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0784282e0aaaff940f9515f44b229c4f93907c9657748d25ae94f59437bac840" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-hpa-724bc-predictor-795997bf5d-xwcpq |
AddedInterface |
Add eth0 [10.133.0.25/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-hpa-724bc-predictor-795997bf5d |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-hpa-724bc-predictor-795997bf5d-xwcpq | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-hpa-724bc-predictor-8485c796b7 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-hpa-724bc-predictor-8485c796b7-nglk8 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-724bc-predictor-795997bf5d-xwcpq |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-724bc-predictor-8485c796b7-nglk8 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-724bc-predictor-8485c796b7-nglk8 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-724bc-predictor-795997bf5d-xwcpq |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-hpa-724bc-predictor-8485c796b7-nglk8 |
AddedInterface |
Add eth0 [10.133.0.26/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-3b643-predictor-94775489-88l6s |
Killing |
Stopping container kserve-container | |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-3b643-predictor-94775489-88l6s |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.24:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-724bc-predictor-795997bf5d-xwcpq |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1326" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-724bc-predictor-795997bf5d-xwcpq |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-724bc-predictor-8485c796b7-nglk8 |
Pulled |
Container image "kserve/xgbserver:latest" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-724bc-predictor-8485c796b7-nglk8 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-724bc-predictor-8485c796b7-nglk8 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-724bc-predictor-795997bf5d-xwcpq |
Created |
Created container: kserve-container | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-724bc-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-724bc-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-724bc-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-724bc-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x7) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-724bc-predictor-8485c796b7-nglk8 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.26:8080: connect: connection refused |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-724bc |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-724bc |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-hpa-724bc] is Ready | |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-724bc-predictor-795997bf5d-xwcpq |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.25:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-724bc |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-724bc |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-hpa-724bc] is Ready | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-724bc-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-724bc-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-724bc-d588cbf89-2tdkd |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:743cff1ceb833e46d1b8e65ba261d1d549e2e90054da6d533b54a9a2f13462c7" already present on machine |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-724bc |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-hpa-724bc] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-hpa-724bc |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-hpa-724bc-d588cbf89 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-hpa-724bc-d588cbf89 |
SuccessfulCreate |
Created pod: model-chainer-raw-hpa-724bc-d588cbf89-2tdkd | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-724bc-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-724bc-d588cbf89-2tdkd |
Created |
Created container: model-chainer-raw-hpa-724bc |
kserve-ci-e2e-test |
multus |
model-chainer-raw-hpa-724bc-d588cbf89-2tdkd |
AddedInterface |
Add eth0 [10.132.0.35/23] from ovn-kubernetes | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-724bc-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-724bc-d588cbf89-2tdkd |
Started |
Started container model-chainer-raw-hpa-724bc |
| (x3) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-724bc-d588cbf89-2tdkd |
BackOff |
Back-off restarting failed container model-chainer-raw-hpa-724bc in pod model-chainer-raw-hpa-724bc-d588cbf89-2tdkd_kserve-ci-e2e-test(44c561b0-a63e-4057-8b7b-792102cd2f53) |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-724bc-predictor-795997bf5d-xwcpq |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-74f70-predictor-767887b594-gh5xm |
AddedInterface |
Add eth0 [10.132.0.36/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-74f70 |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-74f70": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-74f70": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-74f70-predictor-767887b594-gh5xm |
Pulling |
Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" | |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-74f70-predictor-767887b594 |
SuccessfulCreate |
Created pod: message-dumper-raw-74f70-predictor-767887b594-gh5xm | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-724bc-predictor-8485c796b7-nglk8 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-74f70-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-74f70-predictor-767887b594 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-74f70-predictor-767887b594-gh5xm |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-74f70-predictor-767887b594-gh5xm |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-74f70-predictor-767887b594-gh5xm |
Pulled |
Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 939ms (939ms including waiting). Image size: 14813193 bytes. | |
| (x8) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-74f70 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-74f70 |
InferenceServiceReady |
InferenceService [message-dumper-raw-74f70] is Ready | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-74f70-predictor-86f49fcb48 |
SuccessfulCreate |
Created pod: isvc-logger-raw-74f70-predictor-86f49fcb48-hxk5b | |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-74f70-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-74f70-predictor-86f49fcb48 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-74f70 |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-74f70": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-74f70": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-74f70-predictor-86f49fcb48-hxk5b |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-74f70-predictor-86f49fcb48-hxk5b |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-74f70-predictor-86f49fcb48-hxk5b |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0784282e0aaaff940f9515f44b229c4f93907c9657748d25ae94f59437bac840" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-74f70-predictor-86f49fcb48-hxk5b |
AddedInterface |
Add eth0 [10.133.0.27/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-74f70-predictor-86f49fcb48-hxk5b |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:f2fb11757d99ef46b1388e1609ace6b5f516fdebd4ee81c172186c5de9dc18b9" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-74f70-predictor-86f49fcb48-hxk5b |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-74f70-predictor-86f49fcb48-hxk5b |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-74f70-predictor-86f49fcb48-hxk5b |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1326" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-74f70-predictor-86f49fcb48-hxk5b |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-74f70-predictor-86f49fcb48-hxk5b |
Started |
Started container kserve-container | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-74f70-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-74f70-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-74f70-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-74f70-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-74f70-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-74f70-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-74f70 |
InferenceServiceReady |
InferenceService [isvc-logger-raw-74f70] is Ready | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-74f70 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-scale-raw-954a2-predictor-877cc765 |
SuccessfulCreate |
Created pod: isvc-sklearn-scale-raw-954a2-predictor-877cc765-jf86g | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-74f70-predictor-86f49fcb48-hxk5b |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-954a2-predictor-877cc765-jf86g |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0784282e0aaaff940f9515f44b229c4f93907c9657748d25ae94f59437bac840" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-scale-raw-954a2-predictor-877cc765-jf86g |
AddedInterface |
Add eth0 [10.133.0.28/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-74f70-predictor-86f49fcb48-hxk5b |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-74f70-predictor-767887b594-gh5xm |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-scale-raw-954a2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-scale-raw-954a2-predictor-877cc765 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-954a2 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-scale-raw-954a2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scale-raw-954a2": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-954a2-predictor-877cc765-jf86g |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-954a2-predictor-877cc765-jf86g |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-954a2-predictor-877cc765-jf86g |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-954a2-predictor-877cc765-jf86g |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1326" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-954a2-predictor-877cc765-jf86g |
Created |
Created container: kserve-container | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-74f70-predictor-86f49fcb48-hxk5b |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-74f70-predictor-86f49fcb48-hxk5b |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.27:8080: connect: connection refused |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-954a2-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-954a2-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-954a2 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-954a2 |
InferenceServiceReady |
InferenceService [isvc-sklearn-scale-raw-954a2] is Ready | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-89be61 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-89be61": the object has been modified; please apply your changes to the latest version and try again | |
| (x7) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-954a2-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
replicaset-controller |
isvc-primary-89be61-predictor-789bf94c5d |
SuccessfulCreate |
Created pod: isvc-primary-89be61-predictor-789bf94c5d-99btm | |
kserve-ci-e2e-test |
deployment-controller |
isvc-primary-89be61-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-primary-89be61-predictor-789bf94c5d from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-954a2-predictor-877cc765-jf86g |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-89be61 |
UpdateFailed |
Failed to update status for InferenceService "isvc-primary-89be61": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-89be61": the object has been modified; please apply your changes to the latest version and try again | |
| (x7) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-954a2-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-primary-89be61-predictor-789bf94c5d-99btm |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-89be61-predictor-789bf94c5d-99btm |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0784282e0aaaff940f9515f44b229c4f93907c9657748d25ae94f59437bac840" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-89be61-predictor-789bf94c5d-99btm |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-primary-89be61-predictor-789bf94c5d-99btm |
AddedInterface |
Add eth0 [10.133.0.29/23] from ovn-kubernetes | |
| (x16) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-954a2-predictor-877cc765-jf86g |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.28:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-primary-89be61-predictor-789bf94c5d-99btm |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1326" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-89be61-predictor-789bf94c5d-99btm |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-89be61-predictor-789bf94c5d-99btm |
Started |
Started container kserve-container | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-89be61-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-89be61-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-primary-89be61-predictor-789bf94c5d-99btm |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.29:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-89be61 |
InferenceServiceReady |
InferenceService [isvc-primary-89be61] is Ready | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-89be61 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-89be61 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-89be61": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
replicaset-controller |
isvc-secondary-89be61-predictor-5bdc697fb7 |
SuccessfulCreate |
Created pod: isvc-secondary-89be61-predictor-5bdc697fb7-9ndcp | |
kserve-ci-e2e-test |
deployment-controller |
isvc-secondary-89be61-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-secondary-89be61-predictor-5bdc697fb7 from 0 to 1 | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-89be61 |
UpdateFailed |
Failed to update status for InferenceService "isvc-secondary-89be61": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-89be61": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
multus |
isvc-secondary-89be61-predictor-5bdc697fb7-9ndcp |
AddedInterface |
Add eth0 [10.133.0.30/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-89be61-predictor-5bdc697fb7-9ndcp |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0784282e0aaaff940f9515f44b229c4f93907c9657748d25ae94f59437bac840" already present on machine |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-89be61-predictor-5bdc697fb7-9ndcp |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-89be61-predictor-5bdc697fb7-9ndcp |
Started |
Started container storage-initializer |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-89be61-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-89be61-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-89be61-predictor-5bdc697fb7-9ndcp |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-secondary-89be61-predictor-5bdc697fb7-9ndcp_kserve-ci-e2e-test(4e44d3b1-7732-4fc5-81d9-17269abf124a) | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-89be61-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-89be61-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-89be61 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
multus |
isvc-init-fail-b43ebd-predictor-84bd96d595-89tkv |
AddedInterface |
Add eth0 [10.133.0.31/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-init-fail-b43ebd-predictor-84bd96d595 |
SuccessfulCreate |
Created pod: isvc-init-fail-b43ebd-predictor-84bd96d595-89tkv | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-b43ebd |
UpdateFailed |
Failed to update status for InferenceService "isvc-init-fail-b43ebd": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-b43ebd": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-b43ebd |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-b43ebd": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-89be61-predictor-789bf94c5d-99btm |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-init-fail-b43ebd-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-init-fail-b43ebd-predictor-84bd96d595 from 0 to 1 | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-b43ebd-predictor-84bd96d595-89tkv |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-b43ebd-predictor-84bd96d595-89tkv |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-b43ebd-predictor-84bd96d595-89tkv |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0784282e0aaaff940f9515f44b229c4f93907c9657748d25ae94f59437bac840" already present on machine |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-b43ebd |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-5512a-predictor-79fc6d66bc-djd6g |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-5512a-predictor-79fc6d66bc-djd6g |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0784282e0aaaff940f9515f44b229c4f93907c9657748d25ae94f59437bac840" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-5512a-predictor-79fc6d66bc-djd6g |
Created |
Created container: storage-initializer | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-5512a |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-5512a": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-5512a": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
multus |
raw-sklearn-5512a-predictor-79fc6d66bc-djd6g |
AddedInterface |
Add eth0 [10.133.0.32/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-5512a |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-5512a": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-5512a-predictor-79fc6d66bc |
SuccessfulCreate |
Created pod: raw-sklearn-5512a-predictor-79fc6d66bc-djd6g | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-5512a-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-5512a-predictor-79fc6d66bc from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-b43ebd-predictor-84bd96d595-89tkv |
Killing |
Stopping container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-5512a-predictor-79fc6d66bc-djd6g |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1326" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-5512a-predictor-79fc6d66bc-djd6g |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-5512a-predictor-79fc6d66bc-djd6g |
Started |
Started container kserve-container | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-5512a-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-5512a-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x8) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-5512a-predictor-79fc6d66bc-djd6g |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.32:8080: connect: connection refused |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-5512a-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-5512a-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-5512a |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-5512a |
InferenceServiceReady |
InferenceService [raw-sklearn-5512a] is Ready | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-5512a-predictor-79fc6d66bc-djd6g |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-runtime-82bab-predictor-c74656cbf |
SuccessfulCreate |
Created pod: raw-sklearn-runtime-82bab-predictor-c74656cbf-cdn6c | |
kserve-ci-e2e-test |
multus |
raw-sklearn-runtime-82bab-predictor-c74656cbf-cdn6c |
AddedInterface |
Add eth0 [10.133.0.33/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-82bab-predictor-c74656cbf-cdn6c |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0784282e0aaaff940f9515f44b229c4f93907c9657748d25ae94f59437bac840" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-82bab-predictor-c74656cbf-cdn6c |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-82bab-predictor-c74656cbf-cdn6c |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-82bab |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-runtime-82bab": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-82bab": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-runtime-82bab-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-runtime-82bab-predictor-c74656cbf from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-82bab-predictor-c74656cbf-cdn6c |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-82bab-predictor-c74656cbf-cdn6c |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-82bab-predictor-c74656cbf-cdn6c |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1326" already present on machine | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-82bab-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-82bab-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-82bab-predictor-c74656cbf-cdn6c |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.33:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-82bab-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-82bab-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-82bab |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-82bab |
InferenceServiceReady |
InferenceService [raw-sklearn-runtime-82bab] is Ready | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-82bab-predictor-c74656cbf-cdn6c |
Killing |
Stopping container kserve-container |