| Time | Namespace | Component | RelatedObject | Reason | Message |
|---|---|---|---|---|---|
kserve-ci-e2e-test |
model-chainer-raw-c600f-6f845876f-ptr82 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-c600f-6f845876f-ptr82 to ip-10-0-139-205.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-runtime-a6d09-predictor-7dfcbcc5bb-5hvz7 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-runtime-a6d09-predictor-7dfcbcc5bb-5hvz7 to ip-10-0-139-205.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-05c63-predictor-695786cf5d-6qhgw |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-05c63-predictor-695786cf5d-6qhgw to ip-10-0-139-205.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-hpa-78192-7f674fdfb9-dx64t |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-hpa-78192-7f674fdfb9-dx64t to ip-10-0-136-138.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-c600f-predictor-7f8b8cc8fd-4m4wf |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-c600f-predictor-7f8b8cc8fd-4m4wf to ip-10-0-139-205.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-hpa-78192-predictor-774df4866-th9q4 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-78192-predictor-774df4866-th9q4 to ip-10-0-139-205.ec2.internal | ||
kserve-ci-e2e-test |
isvc-secondary-ea9dbb-predictor-647dbdd946-trtm7 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-secondary-ea9dbb-predictor-647dbdd946-trtm7 to ip-10-0-139-205.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-c600f-predictor-695c695d99-nbrm7 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-c600f-predictor-695c695d99-nbrm7 to ip-10-0-139-205.ec2.internal | ||
kserve-ci-e2e-test |
isvc-primary-ea9dbb-predictor-5b757bfb59-xw9mn |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-primary-ea9dbb-predictor-5b757bfb59-xw9mn to ip-10-0-139-205.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-scale-raw-9f1b7-predictor-854f955cf6-r49gz |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-scale-raw-9f1b7-predictor-854f955cf6-r49gz to ip-10-0-139-205.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-hpa-78192-predictor-5dd8cfd66c-l7h2h |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-78192-predictor-5dd8cfd66c-l7h2h to ip-10-0-139-205.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-428b1-predictor-754c8b778b-8vjpm |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-428b1-predictor-754c8b778b-8vjpm to ip-10-0-139-205.ec2.internal | ||
kserve-ci-e2e-test |
isvc-init-fail-2519ca-predictor-bc7f75588-pdl67 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-init-fail-2519ca-predictor-bc7f75588-pdl67 to ip-10-0-139-205.ec2.internal | ||
kserve-ci-e2e-test |
isvc-raw-sklearn-batcher-1ce5e-predictor-8bcd69f4d-xj8t4 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1ce5e-predictor-8bcd69f4d-xj8t4 to ip-10-0-139-205.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-428b1-predictor-795678bc55-ssbzv |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-428b1-predictor-795678bc55-ssbzv to ip-10-0-136-138.ec2.internal | ||
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-1ce5e |
UpdateFailed |
Failed to update status for InferenceService "isvc-raw-sklearn-batcher-1ce5e": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-1ce5e": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-1ce5e-predictor-8bcd69f4d-xj8t4 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:b1408675ea4fdca559a0643f83f1f99c90fa28691ae069983f57f521927df688" | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-1ce5e |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-1ce5e": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
deployment-controller |
isvc-raw-sklearn-batcher-1ce5e-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-raw-sklearn-batcher-1ce5e-predictor-8bcd69f4d from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-raw-sklearn-batcher-1ce5e-predictor-8bcd69f4d |
SuccessfulCreate |
Created pod: isvc-raw-sklearn-batcher-1ce5e-predictor-8bcd69f4d-xj8t4 | |
kserve-ci-e2e-test |
multus |
isvc-raw-sklearn-batcher-1ce5e-predictor-8bcd69f4d-xj8t4 |
AddedInterface |
Add eth0 [10.132.0.36/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-1ce5e-predictor-8bcd69f4d-xj8t4 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-1ce5e-predictor-8bcd69f4d-xj8t4 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-1ce5e-predictor-8bcd69f4d-xj8t4 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:b1408675ea4fdca559a0643f83f1f99c90fa28691ae069983f57f521927df688" in 3.322s (3.322s including waiting). Image size: 299671472 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-1ce5e-predictor-8bcd69f4d-xj8t4 |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1412" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-1ce5e-predictor-8bcd69f4d-xj8t4 |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1412" in 12.531s (12.531s including waiting). Image size: 1560613811 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-1ce5e-predictor-8bcd69f4d-xj8t4 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-1ce5e-predictor-8bcd69f4d-xj8t4 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-1ce5e-predictor-8bcd69f4d-xj8t4 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-agent@sha256:ee4db148793b3f03915a5f82a6e036d7a090e0b557ef078079043d8fab2b15d2" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-1ce5e-predictor-8bcd69f4d-xj8t4 |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-1ce5e-predictor-8bcd69f4d-xj8t4 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:ee4db148793b3f03915a5f82a6e036d7a090e0b557ef078079043d8fab2b15d2" in 1.783s (1.783s including waiting). Image size: 237897299 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-1ce5e-predictor-8bcd69f4d-xj8t4 |
Created |
Created container: agent | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-1ce5e-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-1ce5e-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-1ce5e |
InferenceServiceReady |
InferenceService [isvc-raw-sklearn-batcher-1ce5e] is Ready | |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-1ce5e |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-c600f-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-c600f-predictor-7f8b8cc8fd from 0 to 1 | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-1ce5e-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-c600f-predictor-7f8b8cc8fd |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-c600f-predictor-7f8b8cc8fd-4m4wf | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-1ce5e-predictor-8bcd69f4d-xj8t4 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-c600f-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-c600f-predictor-695c695d99 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-c600f-predictor-695c695d99 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-c600f-predictor-695c695d99-nbrm7 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-c600f |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-c600f": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-c600f": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-1ce5e-predictor-8bcd69f4d-xj8t4 |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-c600f |
UpdateFailed |
Failed to update status for InferenceService "isvc-xgboost-graph-raw-c600f": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-graph-raw-c600f": the object has been modified; please apply your changes to the latest version and try again | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-1ce5e-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-c600f-predictor-7f8b8cc8fd-4m4wf |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-c600f-predictor-7f8b8cc8fd-4m4wf |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:b1408675ea4fdca559a0643f83f1f99c90fa28691ae069983f57f521927df688" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-c600f-predictor-695c695d99-nbrm7 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-c600f-predictor-695c695d99-nbrm7 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-c600f-predictor-695c695d99-nbrm7 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:b1408675ea4fdca559a0643f83f1f99c90fa28691ae069983f57f521927df688" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-c600f-predictor-695c695d99-nbrm7 |
AddedInterface |
Add eth0 [10.132.0.38/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-c600f-predictor-7f8b8cc8fd-4m4wf |
AddedInterface |
Add eth0 [10.132.0.37/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-c600f-predictor-7f8b8cc8fd-4m4wf |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-c600f-predictor-695c695d99-nbrm7 |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-c600f-predictor-7f8b8cc8fd-4m4wf |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-c600f-predictor-7f8b8cc8fd-4m4wf |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-c600f-predictor-7f8b8cc8fd-4m4wf |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1412" already present on machine | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-c600f-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-c600f-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-c600f-predictor-695c695d99-nbrm7 |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 19.199s (19.199s including waiting). Image size: 1306229499 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-c600f-predictor-695c695d99-nbrm7 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-c600f-predictor-695c695d99-nbrm7 |
Created |
Created container: kserve-container | |
| (x12) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-1ce5e-predictor-8bcd69f4d-xj8t4 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.36:8080: connect: connection refused |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-1ce5e-predictor-8bcd69f4d-xj8t4 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-c600f-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-c600f-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-c600f-predictor-7f8b8cc8fd-4m4wf |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.37:8080: connect: connection refused |
| (x7) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-c600f-predictor-695c695d99-nbrm7 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.38:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-c600f |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-c600f |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-c600f] is Ready | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-c600f |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-c600f] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-c600f |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-c600f-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-c600f-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-c600f-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-c600f |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-c600f-6f845876f from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-c600f-6f845876f |
SuccessfulCreate |
Created pod: model-chainer-raw-c600f-6f845876f-ptr82 | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-c600f |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-c600f] is Ready | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-c600f-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
multus |
model-chainer-raw-c600f-6f845876f-ptr82 |
AddedInterface |
Add eth0 [10.132.0.39/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-c600f-6f845876f-ptr82 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-router@sha256:6060da2148ba78c06d6de96222351f1e5f7d3ada4027ac02142aa0e1928e4f29" | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-c600f-6f845876f-ptr82 |
Created |
Created container: model-chainer-raw-c600f | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-c600f-6f845876f-ptr82 |
Started |
Started container model-chainer-raw-c600f | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-c600f-6f845876f-ptr82 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:6060da2148ba78c06d6de96222351f1e5f7d3ada4027ac02142aa0e1928e4f29" in 2.54s (2.54s including waiting). Image size: 216232023 bytes. | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-c600f-6f845876f-ptr82 |
Killing |
Stopping container model-chainer-raw-c600f | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-hpa-78192-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-hpa-78192-predictor-774df4866 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-hpa-78192-predictor-774df4866 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-hpa-78192-predictor-774df4866-th9q4 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-hpa-78192-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-hpa-78192-predictor-5dd8cfd66c from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-hpa-78192-predictor-5dd8cfd66c |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-hpa-78192-predictor-5dd8cfd66c-l7h2h | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-c600f-predictor-7f8b8cc8fd-4m4wf |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-78192-predictor-5dd8cfd66c-l7h2h |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:b1408675ea4fdca559a0643f83f1f99c90fa28691ae069983f57f521927df688" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-78192-predictor-774df4866-th9q4 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-78192-predictor-774df4866-th9q4 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-78192-predictor-774df4866-th9q4 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:b1408675ea4fdca559a0643f83f1f99c90fa28691ae069983f57f521927df688" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-hpa-78192-predictor-774df4866-th9q4 |
AddedInterface |
Add eth0 [10.132.0.40/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-c600f-predictor-695c695d99-nbrm7 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-hpa-78192-predictor-5dd8cfd66c-l7h2h |
AddedInterface |
Add eth0 [10.132.0.41/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-78192-predictor-5dd8cfd66c-l7h2h |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-78192-predictor-5dd8cfd66c-l7h2h |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-78192-predictor-5dd8cfd66c-l7h2h |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-78192-predictor-5dd8cfd66c-l7h2h |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-78192-predictor-774df4866-th9q4 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1412" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-78192-predictor-774df4866-th9q4 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-78192-predictor-774df4866-th9q4 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-78192-predictor-5dd8cfd66c-l7h2h |
Pulled |
Container image "kserve/xgbserver:latest" already present on machine | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-c600f-6f845876f-ptr82 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x7) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-78192-predictor-5dd8cfd66c-l7h2h |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.41:8080: connect: connection refused |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-78192-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-78192-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-78192-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-78192-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-78192-predictor-774df4866-th9q4 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.40:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-78192 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-78192 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-hpa-78192] is Ready | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-78192 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-hpa-78192] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-78192 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-78192-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-78192-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-hpa-78192 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-hpa-78192-7f674fdfb9 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-hpa-78192-7f674fdfb9 |
SuccessfulCreate |
Created pod: model-chainer-raw-hpa-78192-7f674fdfb9-dx64t | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-hpa-78192-7f674fdfb9-dx64t |
AddedInterface |
Add eth0 [10.134.0.19/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-78192-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-78192-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-78192-7f674fdfb9-dx64t |
Pulling |
Pulling image "quay.io/opendatahub/kserve-router@sha256:6060da2148ba78c06d6de96222351f1e5f7d3ada4027ac02142aa0e1928e4f29" | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-78192 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-hpa-78192] is Ready | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-78192-7f674fdfb9-dx64t |
Created |
Created container: model-chainer-raw-hpa-78192 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-78192-7f674fdfb9-dx64t |
Started |
Started container model-chainer-raw-hpa-78192 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-78192-7f674fdfb9-dx64t |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:6060da2148ba78c06d6de96222351f1e5f7d3ada4027ac02142aa0e1928e4f29" in 2.372s (2.372s including waiting). Image size: 216232023 bytes. | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-78192-7f674fdfb9-dx64t |
Killing |
Stopping container model-chainer-raw-hpa-78192 | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-428b1-predictor-795678bc55-ssbzv |
AddedInterface |
Add eth0 [10.134.0.20/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-428b1 |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-428b1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-428b1": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-78192-predictor-774df4866-th9q4 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-428b1-predictor-795678bc55 |
SuccessfulCreate |
Created pod: message-dumper-raw-428b1-predictor-795678bc55-ssbzv | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-428b1-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-428b1-predictor-795678bc55 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-78192-predictor-5dd8cfd66c-l7h2h |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-428b1-predictor-795678bc55-ssbzv |
Pulling |
Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-428b1-predictor-795678bc55-ssbzv |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.20:8080: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-428b1-predictor-795678bc55-ssbzv |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-428b1-predictor-795678bc55-ssbzv |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-428b1-predictor-795678bc55-ssbzv |
Pulled |
Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.061s (1.061s including waiting). Image size: 14813193 bytes. | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-428b1 |
InferenceServiceReady |
InferenceService [message-dumper-raw-428b1] is Ready | |
| (x8) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-428b1 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-428b1-predictor-754c8b778b |
SuccessfulCreate |
Created pod: isvc-logger-raw-428b1-predictor-754c8b778b-8vjpm | |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-428b1-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-428b1-predictor-754c8b778b from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-428b1 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-428b1": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-428b1-predictor-754c8b778b-8vjpm |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-428b1-predictor-754c8b778b-8vjpm |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-428b1-predictor-754c8b778b-8vjpm |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:b1408675ea4fdca559a0643f83f1f99c90fa28691ae069983f57f521927df688" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-428b1-predictor-754c8b778b-8vjpm |
AddedInterface |
Add eth0 [10.132.0.42/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-428b1 |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-428b1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-428b1": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-428b1-predictor-754c8b778b-8vjpm |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:ee4db148793b3f03915a5f82a6e036d7a090e0b557ef078079043d8fab2b15d2" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-428b1-predictor-754c8b778b-8vjpm |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-428b1-predictor-754c8b778b-8vjpm |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-428b1-predictor-754c8b778b-8vjpm |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1412" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-428b1-predictor-754c8b778b-8vjpm |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-428b1-predictor-754c8b778b-8vjpm |
Created |
Created container: kserve-container | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-78192-7f674fdfb9-dx64t |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-428b1-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-428b1-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-428b1-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-428b1-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-428b1-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-428b1-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-428b1 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-428b1 |
InferenceServiceReady |
InferenceService [isvc-logger-raw-428b1] is Ready | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-428b1-predictor-795678bc55-ssbzv |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-428b1-predictor-754c8b778b-8vjpm |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-428b1-predictor-754c8b778b-8vjpm |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-9f1b7 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-scale-raw-9f1b7": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scale-raw-9f1b7": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-scale-raw-9f1b7-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-scale-raw-9f1b7-predictor-854f955cf6 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-scale-raw-9f1b7-predictor-854f955cf6 |
SuccessfulCreate |
Created pod: isvc-sklearn-scale-raw-9f1b7-predictor-854f955cf6-r49gz | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-9f1b7-predictor-854f955cf6-r49gz |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-scale-raw-9f1b7-predictor-854f955cf6-r49gz |
AddedInterface |
Add eth0 [10.132.0.43/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-9f1b7-predictor-854f955cf6-r49gz |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:b1408675ea4fdca559a0643f83f1f99c90fa28691ae069983f57f521927df688" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-9f1b7-predictor-854f955cf6-r49gz |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-9f1b7-predictor-854f955cf6-r49gz |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-9f1b7-predictor-854f955cf6-r49gz |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1412" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-9f1b7-predictor-854f955cf6-r49gz |
Started |
Started container kserve-container | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-428b1-predictor-754c8b778b-8vjpm |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x12) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-428b1-predictor-754c8b778b-8vjpm |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.42:8080: connect: connection refused |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-9f1b7-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-9f1b7-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-9f1b7 |
InferenceServiceReady |
InferenceService [isvc-sklearn-scale-raw-9f1b7] is Ready | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-9f1b7 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-ea9dbb |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-ea9dbb": the object has been modified; please apply your changes to the latest version and try again | |
| (x7) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-9f1b7-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
replicaset-controller |
isvc-primary-ea9dbb-predictor-5b757bfb59 |
SuccessfulCreate |
Created pod: isvc-primary-ea9dbb-predictor-5b757bfb59-xw9mn | |
kserve-ci-e2e-test |
deployment-controller |
isvc-primary-ea9dbb-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-primary-ea9dbb-predictor-5b757bfb59 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-ea9dbb |
UpdateFailed |
Failed to update status for InferenceService "isvc-primary-ea9dbb": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-ea9dbb": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-9f1b7-predictor-854f955cf6-r49gz |
Killing |
Stopping container kserve-container | |
| (x7) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-9f1b7-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
multus |
isvc-primary-ea9dbb-predictor-5b757bfb59-xw9mn |
AddedInterface |
Add eth0 [10.132.0.44/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-ea9dbb-predictor-5b757bfb59-xw9mn |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:b1408675ea4fdca559a0643f83f1f99c90fa28691ae069983f57f521927df688" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-ea9dbb-predictor-5b757bfb59-xw9mn |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-ea9dbb-predictor-5b757bfb59-xw9mn |
Created |
Created container: storage-initializer | |
| (x17) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-9f1b7-predictor-854f955cf6-r49gz |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.43:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-primary-ea9dbb-predictor-5b757bfb59-xw9mn |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-ea9dbb-predictor-5b757bfb59-xw9mn |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-ea9dbb-predictor-5b757bfb59-xw9mn |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1412" already present on machine | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-ea9dbb-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-ea9dbb-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-primary-ea9dbb-predictor-5b757bfb59-xw9mn |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.44:8080: connect: connection refused |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-ea9dbb |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-ea9dbb |
InferenceServiceReady |
InferenceService [isvc-primary-ea9dbb] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
isvc-secondary-ea9dbb-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-secondary-ea9dbb-predictor-647dbdd946 from 0 to 1 | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-ea9dbb |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-ea9dbb": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
multus |
isvc-secondary-ea9dbb-predictor-647dbdd946-trtm7 |
AddedInterface |
Add eth0 [10.132.0.45/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-secondary-ea9dbb-predictor-647dbdd946 |
SuccessfulCreate |
Created pod: isvc-secondary-ea9dbb-predictor-647dbdd946-trtm7 | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-ea9dbb |
UpdateFailed |
Failed to update status for InferenceService "isvc-secondary-ea9dbb": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-ea9dbb": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-ea9dbb-predictor-647dbdd946-trtm7 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:b1408675ea4fdca559a0643f83f1f99c90fa28691ae069983f57f521927df688" already present on machine |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-ea9dbb-predictor-647dbdd946-trtm7 |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-ea9dbb-predictor-647dbdd946-trtm7 |
Started |
Started container storage-initializer |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-ea9dbb-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-ea9dbb-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-ea9dbb-predictor-647dbdd946-trtm7 |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-secondary-ea9dbb-predictor-647dbdd946-trtm7_kserve-ci-e2e-test(d6a3ef60-e456-4952-b71d-0e6c04dc39b4) | |
| (x15) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-ea9dbb |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-ea9dbb-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-ea9dbb-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-2519ca |
UpdateFailed |
Failed to update status for InferenceService "isvc-init-fail-2519ca": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-2519ca": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-init-fail-2519ca-predictor-bc7f75588-pdl67 |
AddedInterface |
Add eth0 [10.132.0.46/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-2519ca |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-2519ca": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-ea9dbb-predictor-5b757bfb59-xw9mn |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-init-fail-2519ca-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-init-fail-2519ca-predictor-bc7f75588 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-init-fail-2519ca-predictor-bc7f75588 |
SuccessfulCreate |
Created pod: isvc-init-fail-2519ca-predictor-bc7f75588-pdl67 | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-2519ca |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-2519ca-predictor-bc7f75588-pdl67 |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-2519ca-predictor-bc7f75588-pdl67 |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-2519ca-predictor-bc7f75588-pdl67 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:b1408675ea4fdca559a0643f83f1f99c90fa28691ae069983f57f521927df688" already present on machine |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-05c63-predictor-695786cf5d-6qhgw |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:b1408675ea4fdca559a0643f83f1f99c90fa28691ae069983f57f521927df688" already present on machine | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-05c63 |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-05c63": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-05c63": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-05c63-predictor-695786cf5d-6qhgw |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-05c63-predictor-695786cf5d-6qhgw |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
raw-sklearn-05c63-predictor-695786cf5d-6qhgw |
AddedInterface |
Add eth0 [10.132.0.47/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-05c63 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-05c63": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-05c63-predictor-695786cf5d |
SuccessfulCreate |
Created pod: raw-sklearn-05c63-predictor-695786cf5d-6qhgw | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-05c63-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-05c63-predictor-695786cf5d from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-2519ca-predictor-bc7f75588-pdl67 |
Killing |
Stopping container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-05c63-predictor-695786cf5d-6qhgw |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1412" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-05c63-predictor-695786cf5d-6qhgw |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-05c63-predictor-695786cf5d-6qhgw |
Started |
Started container kserve-container | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-05c63-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-05c63-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-05c63-predictor-695786cf5d-6qhgw |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.47:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-05c63-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-05c63-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-05c63 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-05c63 |
InferenceServiceReady |
InferenceService [raw-sklearn-05c63] is Ready | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-a6d09 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-a6d09": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-runtime-a6d09-predictor-7dfcbcc5bb |
SuccessfulCreate |
Created pod: raw-sklearn-runtime-a6d09-predictor-7dfcbcc5bb-5hvz7 | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-05c63-predictor-695786cf5d-6qhgw |
Killing |
Stopping container kserve-container | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-a6d09 |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-runtime-a6d09": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-a6d09": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-runtime-a6d09-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-runtime-a6d09-predictor-7dfcbcc5bb from 0 to 1 | |
kserve-ci-e2e-test |
multus |
raw-sklearn-runtime-a6d09-predictor-7dfcbcc5bb-5hvz7 |
AddedInterface |
Add eth0 [10.132.0.48/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-a6d09-predictor-7dfcbcc5bb-5hvz7 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:b1408675ea4fdca559a0643f83f1f99c90fa28691ae069983f57f521927df688" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-a6d09-predictor-7dfcbcc5bb-5hvz7 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-a6d09-predictor-7dfcbcc5bb-5hvz7 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-a6d09-predictor-7dfcbcc5bb-5hvz7 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-a6d09-predictor-7dfcbcc5bb-5hvz7 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-a6d09-predictor-7dfcbcc5bb-5hvz7 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1412" already present on machine | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-a6d09-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-a6d09-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x8) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-a6d09-predictor-7dfcbcc5bb-5hvz7 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.48:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-a6d09-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-a6d09-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x14) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-a6d09 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-a6d09 |
InferenceServiceReady |
InferenceService [raw-sklearn-runtime-a6d09] is Ready | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-a6d09-predictor-7dfcbcc5bb-5hvz7 |
Killing |
Stopping container kserve-container |