| Time | Namespace | Component | RelatedObject | Reason | Message |
|---|---|---|---|---|---|
kserve-ci-e2e-test |
model-chainer-raw-0fa3d-65695c4586-f64qw |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-0fa3d-65695c4586-f64qw to ip-10-0-142-46.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-runtime-d8ecb-predictor-5ffd647596-dr7xt |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-runtime-d8ecb-predictor-5ffd647596-dr7xt to ip-10-0-142-46.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-7eb70-predictor-8458bdd6cb-ntq7p |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-7eb70-predictor-8458bdd6cb-ntq7p to ip-10-0-142-46.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-hpa-007f8-774dcbc4f8-4hnw6 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-hpa-007f8-774dcbc4f8-4hnw6 to ip-10-0-142-46.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-hpa-007f8-predictor-775bddbb67-lhs6k |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-007f8-predictor-775bddbb67-lhs6k to ip-10-0-142-46.ec2.internal | ||
kserve-ci-e2e-test |
isvc-raw-sklearn-batcher-f99a6-predictor-698b5c9d6b-xn5s2 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f99a6-predictor-698b5c9d6b-xn5s2 to ip-10-0-142-46.ec2.internal | ||
kserve-ci-e2e-test |
isvc-secondary-f9fec2-predictor-56fc78594d-hj4lv |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-secondary-f9fec2-predictor-56fc78594d-hj4lv to ip-10-0-142-46.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-hpa-007f8-predictor-578bc4d7bd-7hkkz |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-007f8-predictor-578bc4d7bd-7hkkz to ip-10-0-142-46.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-0fa3d-predictor-5ccf48bd8d-jdt78 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-0fa3d-predictor-5ccf48bd8d-jdt78 to ip-10-0-142-46.ec2.internal | ||
kserve-ci-e2e-test |
isvc-primary-f9fec2-predictor-784f9bbf45-cgzlp |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-primary-f9fec2-predictor-784f9bbf45-cgzlp to ip-10-0-142-46.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-0fa3d-predictor-74f94bbd95-z2d7k |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-0fa3d-predictor-74f94bbd95-z2d7k to ip-10-0-142-46.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-116da-predictor-679df64688-9nfhv |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-116da-predictor-679df64688-9nfhv to ip-10-0-142-46.ec2.internal | ||
kserve-ci-e2e-test |
isvc-init-fail-8c321a-predictor-6cd87d9f66-z7rlh |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-init-fail-8c321a-predictor-6cd87d9f66-z7rlh to ip-10-0-142-46.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-scale-raw-ac40b-predictor-56648db454-vwl25 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-scale-raw-ac40b-predictor-56648db454-vwl25 to ip-10-0-142-46.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-116da-predictor-6578d49f9-hhb9v |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-116da-predictor-6578d49f9-hhb9v to ip-10-0-142-46.ec2.internal | ||
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-f99a6 |
UpdateFailed |
Failed to update status for InferenceService "isvc-raw-sklearn-batcher-f99a6": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-f99a6": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
multus |
isvc-raw-sklearn-batcher-f99a6-predictor-698b5c9d6b-xn5s2 |
AddedInterface |
Add eth0 [10.133.0.18/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
isvc-raw-sklearn-batcher-f99a6-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-raw-sklearn-batcher-f99a6-predictor-698b5c9d6b from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-raw-sklearn-batcher-f99a6-predictor-698b5c9d6b |
SuccessfulCreate |
Created pod: isvc-raw-sklearn-batcher-f99a6-predictor-698b5c9d6b-xn5s2 | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-f99a6 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-f99a6": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-f99a6-predictor-698b5c9d6b-xn5s2 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-f99a6-predictor-698b5c9d6b-xn5s2 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" in 4.475s (4.475s including waiting). Image size: 300960724 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-f99a6-predictor-698b5c9d6b-xn5s2 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-f99a6-predictor-698b5c9d6b-xn5s2 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-f99a6-predictor-698b5c9d6b-xn5s2 |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1407" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-f99a6-predictor-698b5c9d6b-xn5s2 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-f99a6-predictor-698b5c9d6b-xn5s2 |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1407" in 14.082s (14.082s including waiting). Image size: 1560613827 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-f99a6-predictor-698b5c9d6b-xn5s2 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-f99a6-predictor-698b5c9d6b-xn5s2 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-agent@sha256:93a3c88e5cf08b3e8030b29f6d098cd5406a6fd3ad769e6e697c2ae6303a7cab" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-f99a6-predictor-698b5c9d6b-xn5s2 |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-f99a6-predictor-698b5c9d6b-xn5s2 |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-f99a6-predictor-698b5c9d6b-xn5s2 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:93a3c88e5cf08b3e8030b29f6d098cd5406a6fd3ad769e6e697c2ae6303a7cab" in 3.82s (3.82s including waiting). Image size: 237895798 bytes. | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-f99a6-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-f99a6-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x15) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-f99a6 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-f99a6 |
InferenceServiceReady |
InferenceService [isvc-raw-sklearn-batcher-f99a6] is Ready | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-0fa3d-predictor-74f94bbd95 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-0fa3d-predictor-74f94bbd95-z2d7k | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-f99a6-predictor-698b5c9d6b-xn5s2 |
Killing |
Stopping container kserve-container | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-f99a6-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-f99a6-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-0fa3d |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-0fa3d": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-0fa3d": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-0fa3d-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-0fa3d-predictor-74f94bbd95 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-0fa3d |
UpdateFailed |
Failed to update status for InferenceService "isvc-xgboost-graph-raw-0fa3d": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-graph-raw-0fa3d": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-f99a6-predictor-698b5c9d6b-xn5s2 |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-0fa3d-predictor-5ccf48bd8d |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-0fa3d-predictor-5ccf48bd8d-jdt78 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-0fa3d-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-0fa3d-predictor-5ccf48bd8d from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-0fa3d-predictor-74f94bbd95-z2d7k |
AddedInterface |
Add eth0 [10.133.0.20/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-0fa3d-predictor-5ccf48bd8d-jdt78 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-0fa3d-predictor-74f94bbd95-z2d7k |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-0fa3d-predictor-74f94bbd95-z2d7k |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-0fa3d-predictor-74f94bbd95-z2d7k |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-0fa3d-predictor-5ccf48bd8d-jdt78 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-0fa3d-predictor-5ccf48bd8d-jdt78 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-0fa3d-predictor-5ccf48bd8d-jdt78 |
AddedInterface |
Add eth0 [10.133.0.19/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-0fa3d-predictor-74f94bbd95-z2d7k |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-0fa3d-predictor-5ccf48bd8d-jdt78 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-0fa3d-predictor-5ccf48bd8d-jdt78 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-0fa3d-predictor-5ccf48bd8d-jdt78 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1407" already present on machine | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-0fa3d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-0fa3d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-f99a6-predictor-698b5c9d6b-xn5s2 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-f99a6-predictor-698b5c9d6b-xn5s2 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.18:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-0fa3d-predictor-74f94bbd95-z2d7k |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-0fa3d-predictor-74f94bbd95-z2d7k |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-0fa3d-predictor-74f94bbd95-z2d7k |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 21.719s (21.719s including waiting). Image size: 1306229499 bytes. | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-0fa3d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-0fa3d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-0fa3d-predictor-5ccf48bd8d-jdt78 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.19:8080: connect: connection refused |
| (x7) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-0fa3d-predictor-74f94bbd95-z2d7k |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.20:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-0fa3d |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-0fa3d] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-0fa3d |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-0fa3d |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-0fa3d] is Ready | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-0fa3d |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-0fa3d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-0fa3d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-0fa3d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-0fa3d-65695c4586-f64qw |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-raw-0fa3d-serving-cert" not found | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-0fa3d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-0fa3d |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-0fa3d] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-0fa3d |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-0fa3d-65695c4586 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-0fa3d-65695c4586 |
SuccessfulCreate |
Created pod: model-chainer-raw-0fa3d-65695c4586-f64qw | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-0fa3d-65695c4586-f64qw |
AddedInterface |
Add eth0 [10.133.0.21/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-0fa3d-65695c4586-f64qw |
Pulling |
Pulling image "quay.io/opendatahub/kserve-router@sha256:daaaf0b7769b899c00520a668e190161eefaa93b6eb75bfb8161ae18d89812c6" | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-0fa3d-65695c4586-f64qw |
Started |
Started container model-chainer-raw-0fa3d | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-0fa3d-65695c4586-f64qw |
Created |
Created container: model-chainer-raw-0fa3d | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-0fa3d-65695c4586-f64qw |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:daaaf0b7769b899c00520a668e190161eefaa93b6eb75bfb8161ae18d89812c6" in 2.912s (2.912s including waiting). Image size: 216232057 bytes. | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-0fa3d-65695c4586-f64qw |
Killing |
Stopping container model-chainer-raw-0fa3d | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-hpa-007f8-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-hpa-007f8-predictor-578bc4d7bd from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-007f8-predictor-775bddbb67-lhs6k |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-hpa-007f8-predictor-775bddbb67-lhs6k |
AddedInterface |
Add eth0 [10.133.0.23/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-hpa-007f8-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-hpa-007f8-predictor-775bddbb67 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-hpa-007f8-predictor-775bddbb67 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-hpa-007f8-predictor-775bddbb67-lhs6k | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-0fa3d-predictor-5ccf48bd8d-jdt78 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-007f8-predictor-578bc4d7bd-7hkkz |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-007f8-predictor-578bc4d7bd-7hkkz |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-007f8-predictor-578bc4d7bd-7hkkz |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-hpa-007f8-predictor-578bc4d7bd-7hkkz |
AddedInterface |
Add eth0 [10.133.0.22/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-hpa-007f8-predictor-578bc4d7bd |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-hpa-007f8-predictor-578bc4d7bd-7hkkz | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-007f8-predictor-775bddbb67-lhs6k |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-007f8-predictor-775bddbb67-lhs6k |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-0fa3d-predictor-74f94bbd95-z2d7k |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-007f8-predictor-775bddbb67-lhs6k |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-007f8-predictor-578bc4d7bd-7hkkz |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-007f8-predictor-578bc4d7bd-7hkkz |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-007f8-predictor-775bddbb67-lhs6k |
Pulled |
Container image "kserve/xgbserver:latest" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-007f8-predictor-775bddbb67-lhs6k |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-007f8-predictor-578bc4d7bd-7hkkz |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1407" already present on machine | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-007f8-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-007f8-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x7) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-0fa3d-65695c4586-f64qw |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-007f8-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-007f8-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x7) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-007f8-predictor-775bddbb67-lhs6k |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.23:8080: connect: connection refused |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-007f8-predictor-578bc4d7bd-7hkkz |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.22:8080: connect: connection refused |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-007f8 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-007f8 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-hpa-007f8] is Ready | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-007f8 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-hpa-007f8] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-007f8 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
multus |
model-chainer-raw-hpa-007f8-774dcbc4f8-4hnw6 |
AddedInterface |
Add eth0 [10.133.0.24/23] from ovn-kubernetes | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-007f8-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-007f8-774dcbc4f8-4hnw6 |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:daaaf0b7769b899c00520a668e190161eefaa93b6eb75bfb8161ae18d89812c6" already present on machine | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-007f8-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-007f8 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-hpa-007f8] is Ready | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-007f8-774dcbc4f8-4hnw6 |
Started |
Started container model-chainer-raw-hpa-007f8 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-007f8-774dcbc4f8-4hnw6 |
Created |
Created container: model-chainer-raw-hpa-007f8 | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-hpa-007f8 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-hpa-007f8-774dcbc4f8 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-hpa-007f8-774dcbc4f8 |
SuccessfulCreate |
Created pod: model-chainer-raw-hpa-007f8-774dcbc4f8-4hnw6 | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-007f8-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-007f8-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-007f8-predictor-775bddbb67-lhs6k |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-116da-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-116da-predictor-6578d49f9 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-007f8-predictor-578bc4d7bd-7hkkz |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-007f8-774dcbc4f8-4hnw6 |
Killing |
Stopping container model-chainer-raw-hpa-007f8 | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-116da |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-116da": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-116da": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-116da-predictor-6578d49f9-hhb9v |
AddedInterface |
Add eth0 [10.133.0.25/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-116da-predictor-6578d49f9-hhb9v |
Pulling |
Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" | |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-116da-predictor-6578d49f9 |
SuccessfulCreate |
Created pod: message-dumper-raw-116da-predictor-6578d49f9-hhb9v | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-116da-predictor-6578d49f9-hhb9v |
Pulled |
Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.127s (1.127s including waiting). Image size: 14813193 bytes. | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-116da-predictor-6578d49f9-hhb9v |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-116da-predictor-6578d49f9-hhb9v |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-116da |
InferenceServiceReady |
InferenceService [message-dumper-raw-116da] is Ready | |
| (x7) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-116da |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-116da-predictor-679df64688 |
SuccessfulCreate |
Created pod: isvc-logger-raw-116da-predictor-679df64688-9nfhv | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-116da-predictor-679df64688-9nfhv |
Created |
Created container: storage-initializer | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-116da |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-116da": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-116da": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-116da-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-116da-predictor-679df64688 from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-116da-predictor-679df64688-9nfhv |
AddedInterface |
Add eth0 [10.133.0.26/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-116da |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-116da": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-116da-predictor-679df64688-9nfhv |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-116da-predictor-679df64688-9nfhv |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-116da-predictor-679df64688-9nfhv |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-116da-predictor-679df64688-9nfhv |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-116da-predictor-679df64688-9nfhv |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:93a3c88e5cf08b3e8030b29f6d098cd5406a6fd3ad769e6e697c2ae6303a7cab" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-116da-predictor-679df64688-9nfhv |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-116da-predictor-679df64688-9nfhv |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-116da-predictor-679df64688-9nfhv |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1407" already present on machine | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-007f8-774dcbc4f8-4hnw6 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-116da-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-116da-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-116da-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-116da-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-116da-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-116da-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-116da |
InferenceServiceReady |
InferenceService [isvc-logger-raw-116da] is Ready | |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-116da |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-116da-predictor-679df64688-9nfhv |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-scale-raw-ac40b-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-scale-raw-ac40b-predictor-56648db454 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-116da-predictor-6578d49f9-hhb9v |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-ac40b |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-scale-raw-ac40b": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scale-raw-ac40b": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-scale-raw-ac40b-predictor-56648db454 |
SuccessfulCreate |
Created pod: isvc-sklearn-scale-raw-ac40b-predictor-56648db454-vwl25 | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-116da-predictor-679df64688-9nfhv |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-ac40b-predictor-56648db454-vwl25 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-scale-raw-ac40b-predictor-56648db454-vwl25 |
AddedInterface |
Add eth0 [10.133.0.27/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-ac40b-predictor-56648db454-vwl25 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-ac40b-predictor-56648db454-vwl25 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-ac40b-predictor-56648db454-vwl25 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1407" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-ac40b-predictor-56648db454-vwl25 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-ac40b-predictor-56648db454-vwl25 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-ac40b-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-ac40b-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x12) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-116da-predictor-679df64688-9nfhv |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.26:8080: connect: connection refused |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-116da-predictor-679df64688-9nfhv |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-ac40b |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-ac40b |
InferenceServiceReady |
InferenceService [isvc-sklearn-scale-raw-ac40b] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-ac40b-predictor-56648db454-vwl25 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-primary-f9fec2-predictor-784f9bbf45 |
SuccessfulCreate |
Created pod: isvc-primary-f9fec2-predictor-784f9bbf45-cgzlp | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-f9fec2 |
UpdateFailed |
Failed to update status for InferenceService "isvc-primary-f9fec2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-f9fec2": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-f9fec2 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-f9fec2": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-primary-f9fec2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-primary-f9fec2-predictor-784f9bbf45 from 0 to 1 | |
| (x9) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-ac40b-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x9) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-ac40b-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
multus |
isvc-primary-f9fec2-predictor-784f9bbf45-cgzlp |
AddedInterface |
Add eth0 [10.133.0.28/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-f9fec2-predictor-784f9bbf45-cgzlp |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-f9fec2-predictor-784f9bbf45-cgzlp |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-f9fec2-predictor-784f9bbf45-cgzlp |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-f9fec2-predictor-784f9bbf45-cgzlp |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1407" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-f9fec2-predictor-784f9bbf45-cgzlp |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-f9fec2-predictor-784f9bbf45-cgzlp |
Created |
Created container: kserve-container | |
| (x17) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-ac40b-predictor-56648db454-vwl25 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.27:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-f9fec2-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-f9fec2-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-primary-f9fec2-predictor-784f9bbf45-cgzlp |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.28:8080: connect: connection refused |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-f9fec2 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-f9fec2 |
InferenceServiceReady |
InferenceService [isvc-primary-f9fec2] is Ready | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-secondary-f9fec2-predictor-56fc78594d |
SuccessfulCreate |
Created pod: isvc-secondary-f9fec2-predictor-56fc78594d-hj4lv | |
kserve-ci-e2e-test |
deployment-controller |
isvc-secondary-f9fec2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-secondary-f9fec2-predictor-56fc78594d from 0 to 1 | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-f9fec2 |
UpdateFailed |
Failed to update status for InferenceService "isvc-secondary-f9fec2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-f9fec2": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-f9fec2 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-f9fec2": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
multus |
isvc-secondary-f9fec2-predictor-56fc78594d-hj4lv |
AddedInterface |
Add eth0 [10.133.0.29/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-f9fec2-predictor-56fc78594d-hj4lv |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-f9fec2-predictor-56fc78594d-hj4lv |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-f9fec2-predictor-56fc78594d-hj4lv |
Started |
Started container storage-initializer |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-f9fec2-predictor-56fc78594d-hj4lv |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-secondary-f9fec2-predictor-56fc78594d-hj4lv_kserve-ci-e2e-test(71dbcd77-e56b-47d2-9890-a4892849c4b5) | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-f9fec2-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-f9fec2-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x14) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-f9fec2 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-f9fec2-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-f9fec2-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
deployment-controller |
isvc-init-fail-8c321a-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-init-fail-8c321a-predictor-6cd87d9f66 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-init-fail-8c321a-predictor-6cd87d9f66 |
SuccessfulCreate |
Created pod: isvc-init-fail-8c321a-predictor-6cd87d9f66-z7rlh | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-f9fec2-predictor-784f9bbf45-cgzlp |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-8c321a |
UpdateFailed |
Failed to update status for InferenceService "isvc-init-fail-8c321a": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-8c321a": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-8c321a |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-8c321a": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-init-fail-8c321a-predictor-6cd87d9f66-z7rlh |
AddedInterface |
Add eth0 [10.133.0.30/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-8c321a-predictor-6cd87d9f66-z7rlh |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-8c321a-predictor-6cd87d9f66-z7rlh |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-8c321a-predictor-6cd87d9f66-z7rlh |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-8c321a-predictor-6cd87d9f66-z7rlh |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-init-fail-8c321a-predictor-6cd87d9f66-z7rlh_kserve-ci-e2e-test(8ab9a5f0-db9c-4c1b-ba7d-4ab756ac0d5b) | |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-8c321a |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-7eb70-predictor-8458bdd6cb-ntq7p |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-7eb70-predictor-8458bdd6cb |
SuccessfulCreate |
Created pod: raw-sklearn-7eb70-predictor-8458bdd6cb-ntq7p | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-7eb70-predictor-8458bdd6cb-ntq7p |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-7eb70 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-7eb70": the object has been modified; please apply your changes to the latest version and try again | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-7eb70 |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-7eb70": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-7eb70": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-7eb70-predictor-8458bdd6cb-ntq7p |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
raw-sklearn-7eb70-predictor-8458bdd6cb-ntq7p |
AddedInterface |
Add eth0 [10.133.0.31/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-7eb70-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-7eb70-predictor-8458bdd6cb from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-7eb70-predictor-8458bdd6cb-ntq7p |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-7eb70-predictor-8458bdd6cb-ntq7p |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-7eb70-predictor-8458bdd6cb-ntq7p |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1407" already present on machine | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-7eb70-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-7eb70-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-7eb70 |
InferenceServiceReady |
InferenceService [raw-sklearn-7eb70] is Ready | |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-7eb70 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-7eb70-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-7eb70-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-7eb70-predictor-8458bdd6cb-ntq7p |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-runtime-d8ecb-predictor-5ffd647596 |
SuccessfulCreate |
Created pod: raw-sklearn-runtime-d8ecb-predictor-5ffd647596-dr7xt | |
kserve-ci-e2e-test |
multus |
raw-sklearn-runtime-d8ecb-predictor-5ffd647596-dr7xt |
AddedInterface |
Add eth0 [10.133.0.32/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-d8ecb-predictor-5ffd647596-dr7xt |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:c49c84e69ddbb36db782821d4aa3d203f458a00c0c04dd506eb662aae339bb8a" already present on machine | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-d8ecb |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-d8ecb": the object has been modified; please apply your changes to the latest version and try again | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-d8ecb |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-runtime-d8ecb": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-d8ecb": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-runtime-d8ecb-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-runtime-d8ecb-predictor-5ffd647596 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-d8ecb-predictor-5ffd647596-dr7xt |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-d8ecb-predictor-5ffd647596-dr7xt |
Started |
Started container storage-initializer | |
| (x9) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-7eb70-predictor-8458bdd6cb-ntq7p |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.31:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-d8ecb-predictor-5ffd647596-dr7xt |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1407" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-d8ecb-predictor-5ffd647596-dr7xt |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-d8ecb-predictor-5ffd647596-dr7xt |
Created |
Created container: kserve-container | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-d8ecb-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-d8ecb-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-d8ecb-predictor-5ffd647596-dr7xt |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.32:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-d8ecb-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-d8ecb-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x14) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-d8ecb |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-d8ecb |
InferenceServiceReady |
InferenceService [raw-sklearn-runtime-d8ecb] is Ready | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-d8ecb-predictor-5ffd647596-dr7xt |
Killing |
Stopping container kserve-container |