| Time | Namespace | Component | RelatedObject | Reason | Message |
|---|---|---|---|---|---|
kserve-ci-e2e-test |
isvc-sklearn-scale-raw-45df6-predictor-854b479db7-7rn5j |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-scale-raw-45df6-predictor-854b479db7-7rn5j to ip-10-0-128-239.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-81bcf-predictor-745f867bd5-jj65m |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-81bcf-predictor-745f867bd5-jj65m to ip-10-0-128-239.ec2.internal | ||
kserve-ci-e2e-test |
isvc-primary-13f916-predictor-7f4b544bb-7sn97 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-primary-13f916-predictor-7f4b544bb-7sn97 to ip-10-0-128-239.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-d0c94-predictor-68c744c676-9sbm5 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-d0c94-predictor-68c744c676-9sbm5 to ip-10-0-135-72.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-d0c94-predictor-84ddff5b74-vs496 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-d0c94-predictor-84ddff5b74-vs496 to ip-10-0-128-239.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-hpa-3e39c-predictor-57c8568889-x96mr |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3e39c-predictor-57c8568889-x96mr to ip-10-0-128-239.ec2.internal | ||
kserve-ci-e2e-test |
isvc-secondary-13f916-predictor-d9cf89fc5-skn6q |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-secondary-13f916-predictor-d9cf89fc5-skn6q to ip-10-0-128-239.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-81bcf-f88d6597b-xj4kr |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-81bcf-f88d6597b-xj4kr to ip-10-0-128-239.ec2.internal | ||
kserve-ci-e2e-test |
isvc-raw-sklearn-batcher-586f1-predictor-55776767cf-j6t2m |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-raw-sklearn-batcher-586f1-predictor-55776767cf-j6t2m to ip-10-0-128-239.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-81bcf-predictor-65c8ddb5cf-7nlsx |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-81bcf-predictor-65c8ddb5cf-7nlsx to ip-10-0-128-239.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-2bb0d-predictor-6755c856c-4flnf |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-2bb0d-predictor-6755c856c-4flnf to ip-10-0-128-239.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-runtime-e7bec-predictor-7ff77696d9-5jjbh |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-runtime-e7bec-predictor-7ff77696d9-5jjbh to ip-10-0-128-239.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-hpa-3e39c-565d6b7c6d-5xzs4 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-hpa-3e39c-565d6b7c6d-5xzs4 to ip-10-0-128-239.ec2.internal | ||
kserve-ci-e2e-test |
isvc-init-fail-dcfe87-predictor-5b4466c647-ndp8h |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-init-fail-dcfe87-predictor-5b4466c647-ndp8h to ip-10-0-128-239.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-hpa-3e39c-predictor-7cf7df947-6rttt |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3e39c-predictor-7cf7df947-6rttt to ip-10-0-128-239.ec2.internal | ||
kserve-ci-e2e-test |
replicaset-controller |
isvc-raw-sklearn-batcher-586f1-predictor-55776767cf |
SuccessfulCreate |
Created pod: isvc-raw-sklearn-batcher-586f1-predictor-55776767cf-j6t2m | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-586f1 |
UpdateFailed |
Failed to update status for InferenceService "isvc-raw-sklearn-batcher-586f1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-586f1": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-586f1 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-586f1": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-raw-sklearn-batcher-586f1-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-raw-sklearn-batcher-586f1-predictor-55776767cf from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-raw-sklearn-batcher-586f1-predictor-55776767cf-j6t2m |
AddedInterface |
Add eth0 [10.133.0.16/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-586f1-predictor-55776767cf-j6t2m |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:a994fd91064f88b0a52b0b466ad083e0715e86891e056482a61038ab37cc2482" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-586f1-predictor-55776767cf-j6t2m |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-586f1-predictor-55776767cf-j6t2m |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-586f1-predictor-55776767cf-j6t2m |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:a994fd91064f88b0a52b0b466ad083e0715e86891e056482a61038ab37cc2482" in 4.476s (4.476s including waiting). Image size: 300957093 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-586f1-predictor-55776767cf-j6t2m |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1435" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-586f1-predictor-55776767cf-j6t2m |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1435" in 14.076s (14.076s including waiting). Image size: 1562514860 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-586f1-predictor-55776767cf-j6t2m |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-586f1-predictor-55776767cf-j6t2m |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-586f1-predictor-55776767cf-j6t2m |
Pulling |
Pulling image "quay.io/opendatahub/kserve-agent@sha256:84e96718cb727bdb7a70c88a1b15da870dac18fe5721015f96b729224acb0c47" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-586f1-predictor-55776767cf-j6t2m |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:84e96718cb727bdb7a70c88a1b15da870dac18fe5721015f96b729224acb0c47" in 2.638s (2.638s including waiting). Image size: 237896263 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-586f1-predictor-55776767cf-j6t2m |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-586f1-predictor-55776767cf-j6t2m |
Started |
Started container agent | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-586f1-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-586f1-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-586f1 |
InferenceServiceReady |
InferenceService [isvc-raw-sklearn-batcher-586f1] is Ready | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-586f1 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-586f1-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-586f1-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-586f1-predictor-55776767cf-j6t2m |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-81bcf-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-81bcf-predictor-745f867bd5 from 0 to 1 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-81bcf-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-81bcf-predictor-65c8ddb5cf from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-81bcf-predictor-65c8ddb5cf |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-81bcf-predictor-65c8ddb5cf-7nlsx | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-81bcf-predictor-745f867bd5 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-81bcf-predictor-745f867bd5-jj65m | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-81bcf-predictor-745f867bd5-jj65m |
AddedInterface |
Add eth0 [10.133.0.17/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-81bcf |
UpdateFailed |
Failed to update status for InferenceService "isvc-xgboost-graph-raw-81bcf": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-graph-raw-81bcf": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-81bcf |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-81bcf": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-81bcf": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-586f1-predictor-55776767cf-j6t2m |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-81bcf-predictor-745f867bd5-jj65m |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a994fd91064f88b0a52b0b466ad083e0715e86891e056482a61038ab37cc2482" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-81bcf-predictor-65c8ddb5cf-7nlsx |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-81bcf-predictor-65c8ddb5cf-7nlsx |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-81bcf-predictor-65c8ddb5cf-7nlsx |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a994fd91064f88b0a52b0b466ad083e0715e86891e056482a61038ab37cc2482" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-81bcf-predictor-65c8ddb5cf-7nlsx |
AddedInterface |
Add eth0 [10.133.0.18/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-81bcf-predictor-745f867bd5-jj65m |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-81bcf-predictor-745f867bd5-jj65m |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-81bcf-predictor-745f867bd5-jj65m |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1435" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-81bcf-predictor-745f867bd5-jj65m |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-81bcf-predictor-745f867bd5-jj65m |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-81bcf-predictor-65c8ddb5cf-7nlsx |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-81bcf-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-81bcf-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-586f1-predictor-55776767cf-j6t2m |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.16:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-586f1-predictor-55776767cf-j6t2m |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-81bcf-predictor-65c8ddb5cf-7nlsx |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 21.208s (21.208s including waiting). Image size: 1306326779 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-81bcf-predictor-65c8ddb5cf-7nlsx |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-81bcf-predictor-65c8ddb5cf-7nlsx |
Created |
Created container: kserve-container | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-81bcf-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-81bcf-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-81bcf-predictor-745f867bd5-jj65m |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.17:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-81bcf |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-81bcf] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-81bcf |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x7) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-81bcf-predictor-65c8ddb5cf-7nlsx |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.18:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-81bcf |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-81bcf] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-81bcf |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-81bcf-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-81bcf-f88d6597b-xj4kr |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-raw-81bcf-serving-cert" not found | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-81bcf |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-81bcf-f88d6597b from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-81bcf-f88d6597b |
SuccessfulCreate |
Created pod: model-chainer-raw-81bcf-f88d6597b-xj4kr | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-81bcf |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-81bcf] is Ready | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-81bcf-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-81bcf-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-81bcf-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-81bcf-f88d6597b-xj4kr |
Pulling |
Pulling image "quay.io/opendatahub/kserve-router@sha256:c077214438a2f9bdb936ff2da330f1b072e635107d34dd2fdb9da07fac4424c5" | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-81bcf-f88d6597b-xj4kr |
AddedInterface |
Add eth0 [10.133.0.19/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-81bcf-f88d6597b-xj4kr |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:c077214438a2f9bdb936ff2da330f1b072e635107d34dd2fdb9da07fac4424c5" in 2.384s (2.384s including waiting). Image size: 216232519 bytes. | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-81bcf-f88d6597b-xj4kr |
Started |
Started container model-chainer-raw-81bcf | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-81bcf-f88d6597b-xj4kr |
Created |
Created container: model-chainer-raw-81bcf | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-81bcf-f88d6597b-xj4kr |
Killing |
Stopping container model-chainer-raw-81bcf | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-hpa-3e39c-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-hpa-3e39c-predictor-57c8568889 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-hpa-3e39c-predictor-57c8568889 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-hpa-3e39c-predictor-57c8568889-x96mr | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-81bcf-predictor-745f867bd5-jj65m |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-hpa-3e39c-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-hpa-3e39c-predictor-7cf7df947 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-hpa-3e39c-predictor-7cf7df947 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-hpa-3e39c-predictor-7cf7df947-6rttt | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-81bcf-predictor-65c8ddb5cf-7nlsx |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3e39c-predictor-7cf7df947-6rttt |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a994fd91064f88b0a52b0b466ad083e0715e86891e056482a61038ab37cc2482" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3e39c-predictor-57c8568889-x96mr |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a994fd91064f88b0a52b0b466ad083e0715e86891e056482a61038ab37cc2482" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-hpa-3e39c-predictor-57c8568889-x96mr |
AddedInterface |
Add eth0 [10.133.0.20/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3e39c-predictor-57c8568889-x96mr |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3e39c-predictor-57c8568889-x96mr |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-hpa-3e39c-predictor-7cf7df947-6rttt |
AddedInterface |
Add eth0 [10.133.0.21/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3e39c-predictor-7cf7df947-6rttt |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3e39c-predictor-7cf7df947-6rttt |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3e39c-predictor-7cf7df947-6rttt |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3e39c-predictor-57c8568889-x96mr |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3e39c-predictor-57c8568889-x96mr |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1435" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3e39c-predictor-57c8568889-x96mr |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3e39c-predictor-7cf7df947-6rttt |
Pulled |
Container image "kserve/xgbserver:latest" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3e39c-predictor-7cf7df947-6rttt |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-3e39c-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-3e39c-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-3e39c-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-3e39c-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-81bcf-f88d6597b-xj4kr |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-81bcf-f88d6597b-xj4kr |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.19:8080/readyz": dial tcp 10.133.0.19:8080: connect: connection refused | |
| (x7) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3e39c-predictor-7cf7df947-6rttt |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.21:8080: connect: connection refused |
| (x7) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3e39c-predictor-57c8568889-x96mr |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.20:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-3e39c |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-hpa-3e39c] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-3e39c |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-3e39c |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-3e39c |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-hpa-3e39c] is Ready | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-3e39c-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-3e39c-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-3e39c-565d6b7c6d-5xzs4 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-raw-hpa-3e39c-serving-cert" not found | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-3e39c |
InternalError |
fails to update InferenceGraph status: Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "model-chainer-raw-hpa-3e39c": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-3e39c |
UpdateFailed |
Failed to update status for InferenceGraph "model-chainer-raw-hpa-3e39c": Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "model-chainer-raw-hpa-3e39c": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-3e39c |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-hpa-3e39c] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-hpa-3e39c |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-hpa-3e39c-565d6b7c6d from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-hpa-3e39c-565d6b7c6d |
SuccessfulCreate |
Created pod: model-chainer-raw-hpa-3e39c-565d6b7c6d-5xzs4 | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-hpa-3e39c-565d6b7c6d-5xzs4 |
AddedInterface |
Add eth0 [10.133.0.22/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-3e39c-565d6b7c6d-5xzs4 |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:c077214438a2f9bdb936ff2da330f1b072e635107d34dd2fdb9da07fac4424c5" already present on machine | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-3e39c-565d6b7c6d-5xzs4 |
Created |
Created container: model-chainer-raw-hpa-3e39c | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-3e39c-565d6b7c6d-5xzs4 |
Started |
Started container model-chainer-raw-hpa-3e39c | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-d0c94-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-d0c94-predictor-68c744c676 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3e39c-predictor-7cf7df947-6rttt |
Killing |
Stopping container kserve-container | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-3e39c-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-3e39c-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-3e39c-565d6b7c6d-5xzs4 |
Killing |
Stopping container model-chainer-raw-hpa-3e39c | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-d0c94-predictor-68c744c676-9sbm5 |
Pulling |
Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-d0c94-predictor-68c744c676-9sbm5 |
AddedInterface |
Add eth0 [10.132.0.34/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3e39c-predictor-57c8568889-x96mr |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-d0c94-predictor-68c744c676 |
SuccessfulCreate |
Created pod: message-dumper-raw-d0c94-predictor-68c744c676-9sbm5 | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-d0c94 |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-d0c94": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-d0c94": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-d0c94 |
InferenceServiceReady |
InferenceService [message-dumper-raw-d0c94] is Ready | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-d0c94-predictor-68c744c676-9sbm5 |
Pulled |
Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 967ms (967ms including waiting). Image size: 14813193 bytes. | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-d0c94-predictor-68c744c676-9sbm5 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-d0c94-predictor-68c744c676-9sbm5 |
Started |
Started container kserve-container | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-d0c94 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d0c94-predictor-84ddff5b74-vs496 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-d0c94-predictor-84ddff5b74-vs496 |
AddedInterface |
Add eth0 [10.133.0.23/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-d0c94 |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-d0c94": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-d0c94": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-d0c94-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-d0c94-predictor-84ddff5b74 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-d0c94-predictor-84ddff5b74 |
SuccessfulCreate |
Created pod: isvc-logger-raw-d0c94-predictor-84ddff5b74-vs496 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-d0c94 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-d0c94": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d0c94-predictor-84ddff5b74-vs496 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d0c94-predictor-84ddff5b74-vs496 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a994fd91064f88b0a52b0b466ad083e0715e86891e056482a61038ab37cc2482" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d0c94-predictor-84ddff5b74-vs496 |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d0c94-predictor-84ddff5b74-vs496 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d0c94-predictor-84ddff5b74-vs496 |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:84e96718cb727bdb7a70c88a1b15da870dac18fe5721015f96b729224acb0c47" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d0c94-predictor-84ddff5b74-vs496 |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d0c94-predictor-84ddff5b74-vs496 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-d0c94-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-d0c94-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d0c94-predictor-84ddff5b74-vs496 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1435" already present on machine | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-3e39c-565d6b7c6d-5xzs4 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-d0c94-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-d0c94-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-d0c94-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-d0c94-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-d0c94 |
InferenceServiceReady |
InferenceService [isvc-logger-raw-d0c94] is Ready | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-d0c94 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-d0c94-predictor-68c744c676-9sbm5 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d0c94-predictor-84ddff5b74-vs496 |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-scale-raw-45df6-predictor-854b479db7 |
SuccessfulCreate |
Created pod: isvc-sklearn-scale-raw-45df6-predictor-854b479db7-7rn5j | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-scale-raw-45df6-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-scale-raw-45df6-predictor-854b479db7 from 0 to 1 | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-d0c94-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-d0c94-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-45df6-predictor-854b479db7-7rn5j |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-scale-raw-45df6-predictor-854b479db7-7rn5j |
AddedInterface |
Add eth0 [10.133.0.24/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-45df6-predictor-854b479db7-7rn5j |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a994fd91064f88b0a52b0b466ad083e0715e86891e056482a61038ab37cc2482" already present on machine | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-45df6 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-scale-raw-45df6": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scale-raw-45df6": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-45df6-predictor-854b479db7-7rn5j |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d0c94-predictor-84ddff5b74-vs496 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-45df6-predictor-854b479db7-7rn5j |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-45df6-predictor-854b479db7-7rn5j |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1435" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-45df6-predictor-854b479db7-7rn5j |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-45df6-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-45df6-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d0c94-predictor-84ddff5b74-vs496 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.23:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d0c94-predictor-84ddff5b74-vs496 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-45df6 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-45df6 |
InferenceServiceReady |
InferenceService [isvc-sklearn-scale-raw-45df6] is Ready | |
| (x8) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-45df6-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x8) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-45df6-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-13f916 |
UpdateFailed |
Failed to update status for InferenceService "isvc-primary-13f916": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-13f916": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-primary-13f916-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-primary-13f916-predictor-7f4b544bb from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-primary-13f916-predictor-7f4b544bb-7sn97 |
AddedInterface |
Add eth0 [10.133.0.25/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-primary-13f916-predictor-7f4b544bb |
SuccessfulCreate |
Created pod: isvc-primary-13f916-predictor-7f4b544bb-7sn97 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-13f916 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-13f916": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-45df6-predictor-854b479db7-7rn5j |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-13f916-predictor-7f4b544bb-7sn97 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-13f916-predictor-7f4b544bb-7sn97 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-13f916-predictor-7f4b544bb-7sn97 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a994fd91064f88b0a52b0b466ad083e0715e86891e056482a61038ab37cc2482" already present on machine | |
| (x15) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-45df6-predictor-854b479db7-7rn5j |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.24:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-primary-13f916-predictor-7f4b544bb-7sn97 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-13f916-predictor-7f4b544bb-7sn97 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1435" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-13f916-predictor-7f4b544bb-7sn97 |
Started |
Started container kserve-container | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-13f916-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-13f916-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-13f916 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-13f916 |
InferenceServiceReady |
InferenceService [isvc-primary-13f916] is Ready | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-13f916 |
UpdateFailed |
Failed to update status for InferenceService "isvc-secondary-13f916": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-13f916": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-13f916 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-13f916": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-secondary-13f916-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-secondary-13f916-predictor-d9cf89fc5 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-secondary-13f916-predictor-d9cf89fc5 |
SuccessfulCreate |
Created pod: isvc-secondary-13f916-predictor-d9cf89fc5-skn6q | |
kserve-ci-e2e-test |
multus |
isvc-secondary-13f916-predictor-d9cf89fc5-skn6q |
AddedInterface |
Add eth0 [10.133.0.26/23] from ovn-kubernetes | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-13f916-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-13f916-predictor-d9cf89fc5-skn6q |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-13f916-predictor-d9cf89fc5-skn6q |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a994fd91064f88b0a52b0b466ad083e0715e86891e056482a61038ab37cc2482" already present on machine |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-13f916-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-13f916-predictor-d9cf89fc5-skn6q |
Started |
Started container storage-initializer |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-13f916-predictor-d9cf89fc5-skn6q |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-secondary-13f916-predictor-d9cf89fc5-skn6q_kserve-ci-e2e-test(e98535e9-0e7b-41a9-b598-b082c4c7d137) | |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-13f916 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-13f916-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-13f916-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-init-fail-dcfe87-predictor-5b4466c647 |
SuccessfulCreate |
Created pod: isvc-init-fail-dcfe87-predictor-5b4466c647-ndp8h | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-dcfe87 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-dcfe87": the object has been modified; please apply your changes to the latest version and try again | |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-primary-13f916-predictor-7f4b544bb-7sn97 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.25:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-primary-13f916-predictor-7f4b544bb-7sn97 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-init-fail-dcfe87-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-init-fail-dcfe87-predictor-5b4466c647 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-dcfe87 |
UpdateFailed |
Failed to update status for InferenceService "isvc-init-fail-dcfe87": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-dcfe87": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-init-fail-dcfe87-predictor-5b4466c647-ndp8h |
AddedInterface |
Add eth0 [10.133.0.27/23] from ovn-kubernetes | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-dcfe87 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-2bb0d-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-2bb0d-predictor-6755c856c from 0 to 1 | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-dcfe87-predictor-5b4466c647-ndp8h |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-dcfe87-predictor-5b4466c647-ndp8h |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a994fd91064f88b0a52b0b466ad083e0715e86891e056482a61038ab37cc2482" already present on machine |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-dcfe87-predictor-5b4466c647-ndp8h |
Created |
Created container: storage-initializer |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-2bb0d-predictor-6755c856c |
SuccessfulCreate |
Created pod: raw-sklearn-2bb0d-predictor-6755c856c-4flnf | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-2bb0d |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-2bb0d": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-2bb0d": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
raw-sklearn-2bb0d-predictor-6755c856c-4flnf |
AddedInterface |
Add eth0 [10.133.0.28/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-2bb0d-predictor-6755c856c-4flnf |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-2bb0d-predictor-6755c856c-4flnf |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-2bb0d-predictor-6755c856c-4flnf |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a994fd91064f88b0a52b0b466ad083e0715e86891e056482a61038ab37cc2482" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-dcfe87-predictor-5b4466c647-ndp8h |
Killing |
Stopping container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-2bb0d-predictor-6755c856c-4flnf |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1435" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-2bb0d-predictor-6755c856c-4flnf |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-2bb0d-predictor-6755c856c-4flnf |
Started |
Started container kserve-container | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-2bb0d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-2bb0d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x7) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-2bb0d-predictor-6755c856c-4flnf |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.28:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-2bb0d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-2bb0d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-2bb0d |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-2bb0d |
InferenceServiceReady |
InferenceService [raw-sklearn-2bb0d] is Ready | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-2bb0d-predictor-6755c856c-4flnf |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
multus |
raw-sklearn-runtime-e7bec-predictor-7ff77696d9-5jjbh |
AddedInterface |
Add eth0 [10.133.0.29/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-e7bec-predictor-7ff77696d9-5jjbh |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:a994fd91064f88b0a52b0b466ad083e0715e86891e056482a61038ab37cc2482" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-e7bec-predictor-7ff77696d9-5jjbh |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-e7bec-predictor-7ff77696d9-5jjbh |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-e7bec |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-runtime-e7bec": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-e7bec": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-runtime-e7bec-predictor-7ff77696d9 |
SuccessfulCreate |
Created pod: raw-sklearn-runtime-e7bec-predictor-7ff77696d9-5jjbh | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-runtime-e7bec-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-runtime-e7bec-predictor-7ff77696d9 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-e7bec-predictor-7ff77696d9-5jjbh |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-e7bec-predictor-7ff77696d9-5jjbh |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-e7bec-predictor-7ff77696d9-5jjbh |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1435" already present on machine | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-e7bec-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-e7bec-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x7) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-e7bec-predictor-7ff77696d9-5jjbh |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.29:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-e7bec-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-e7bec-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-e7bec |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-e7bec |
InferenceServiceReady |
InferenceService [raw-sklearn-runtime-e7bec] is Ready | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-e7bec-predictor-7ff77696d9-5jjbh |
Killing |
Stopping container kserve-container |