| Time | Namespace | Component | RelatedObject | Reason | Message |
|---|---|---|---|---|---|
kserve-ci-e2e-test |
isvc-raw-sklearn-batcher-6af14-predictor-848cbf7b6b-2kdkd |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-raw-sklearn-batcher-6af14-predictor-848cbf7b6b-2kdkd to ip-10-0-141-131.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-hpa-1d91c-predictor-76b8b86b6d-vpcm5 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1d91c-predictor-76b8b86b6d-vpcm5 to ip-10-0-141-131.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-eca71-predictor-df48fb76-48pcr |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-eca71-predictor-df48fb76-48pcr to ip-10-0-141-131.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz to ip-10-0-136-114.ec2.internal | ||
kserve-ci-e2e-test |
isvc-init-fail-9f5836-predictor-585f8bcf78-z24bn |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-init-fail-9f5836-predictor-585f8bcf78-z24bn to ip-10-0-141-131.ec2.internal | ||
kserve-ci-e2e-test |
isvc-secondary-0dd324-predictor-66c78d4b56-8zdcw |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-secondary-0dd324-predictor-66c78d4b56-8zdcw to ip-10-0-141-131.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-bd7db-predictor-556ff547f5-4ms9v |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-bd7db-predictor-556ff547f5-4ms9v to ip-10-0-141-131.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-bd7db-78d659848c-xmbs5 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5 to ip-10-0-136-114.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-hpa-1d91c-predictor-779d987f87-tntcw |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1d91c-predictor-779d987f87-tntcw to ip-10-0-141-131.ec2.internal | ||
kserve-ci-e2e-test |
isvc-primary-0dd324-predictor-5f56f6dd6f-wvg7z |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-primary-0dd324-predictor-5f56f6dd6f-wvg7z to ip-10-0-141-131.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-scale-raw-9b575-predictor-6fd5844c66-whqrv |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-scale-raw-9b575-predictor-6fd5844c66-whqrv to ip-10-0-141-131.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-7cc0c-predictor-7b494555d9-95ltg |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-7cc0c-predictor-7b494555d9-95ltg to ip-10-0-141-131.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-runtime-16ff5-predictor-d968b69d4-q7889 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-runtime-16ff5-predictor-d968b69d4-q7889 to ip-10-0-141-131.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549 to ip-10-0-136-114.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-bd7db-predictor-fd55b8664-sg94q |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-bd7db-predictor-fd55b8664-sg94q to ip-10-0-141-131.ec2.internal | ||
kserve-ci-e2e-test |
deployment-controller |
isvc-raw-sklearn-batcher-6af14-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-raw-sklearn-batcher-6af14-predictor-848cbf7b6b from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-6af14 |
UpdateFailed |
Failed to update status for InferenceService "isvc-raw-sklearn-batcher-6af14": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-6af14": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-raw-sklearn-batcher-6af14-predictor-848cbf7b6b |
SuccessfulCreate |
Created pod: isvc-raw-sklearn-batcher-6af14-predictor-848cbf7b6b-2kdkd | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-6af14 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-6af14": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-6af14-predictor-848cbf7b6b-2kdkd |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:16546eaa530f8a9a22c60a7ffa82a496f33cd9dfcf4989c8baef0968e8392589" | |
kserve-ci-e2e-test |
multus |
isvc-raw-sklearn-batcher-6af14-predictor-848cbf7b6b-2kdkd |
AddedInterface |
Add eth0 [10.132.0.24/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-6af14-predictor-848cbf7b6b-2kdkd |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:16546eaa530f8a9a22c60a7ffa82a496f33cd9dfcf4989c8baef0968e8392589" in 4.607s (4.607s including waiting). Image size: 300955052 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-6af14-predictor-848cbf7b6b-2kdkd |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-6af14-predictor-848cbf7b6b-2kdkd |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-6af14-predictor-848cbf7b6b-2kdkd |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1362" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-6af14-predictor-848cbf7b6b-2kdkd |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1362" in 13.59s (13.59s including waiting). Image size: 1560613810 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-6af14-predictor-848cbf7b6b-2kdkd |
Pulling |
Pulling image "quay.io/opendatahub/kserve-agent@sha256:cda273db4b40777305d78f4b7f8bde558e921e8f875775edae595cda2e7cfa4b" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-6af14-predictor-848cbf7b6b-2kdkd |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-6af14-predictor-848cbf7b6b-2kdkd |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-6af14-predictor-848cbf7b6b-2kdkd |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-6af14-predictor-848cbf7b6b-2kdkd |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-6af14-predictor-848cbf7b6b-2kdkd |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:cda273db4b40777305d78f4b7f8bde558e921e8f875775edae595cda2e7cfa4b" in 2.585s (2.585s including waiting). Image size: 237894732 bytes. | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-6af14-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-6af14-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-6af14 |
InferenceServiceReady |
InferenceService [isvc-raw-sklearn-batcher-6af14] is Ready | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-6af14 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-6af14-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-6af14-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-bd7db |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-bd7db": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-bd7db": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-6af14-predictor-848cbf7b6b-2kdkd |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-6af14-predictor-848cbf7b6b-2kdkd |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-bd7db-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-bd7db-predictor-fd55b8664 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-bd7db-predictor-556ff547f5 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-bd7db-predictor-556ff547f5-4ms9v | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-bd7db-predictor-fd55b8664-sg94q |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-bd7db-predictor-fd55b8664 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-bd7db-predictor-fd55b8664-sg94q | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-bd7db-predictor-fd55b8664-sg94q |
AddedInterface |
Add eth0 [10.132.0.25/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-bd7db-predictor-fd55b8664-sg94q |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-bd7db-predictor-fd55b8664-sg94q |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:16546eaa530f8a9a22c60a7ffa82a496f33cd9dfcf4989c8baef0968e8392589" already present on machine | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-bd7db |
UpdateFailed |
Failed to update status for InferenceService "isvc-xgboost-graph-raw-bd7db": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-graph-raw-bd7db": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-bd7db-predictor-556ff547f5-4ms9v |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-bd7db-predictor-556ff547f5-4ms9v |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-bd7db-predictor-556ff547f5-4ms9v |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:16546eaa530f8a9a22c60a7ffa82a496f33cd9dfcf4989c8baef0968e8392589" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-bd7db-predictor-556ff547f5-4ms9v |
AddedInterface |
Add eth0 [10.132.0.26/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-bd7db-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-bd7db-predictor-556ff547f5 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-bd7db-predictor-fd55b8664-sg94q |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-bd7db-predictor-fd55b8664-sg94q |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-bd7db-predictor-556ff547f5-4ms9v |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-bd7db-predictor-fd55b8664-sg94q |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1362" already present on machine | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-6af14-predictor-848cbf7b6b-2kdkd |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-6af14-predictor-848cbf7b6b-2kdkd |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.24:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-bd7db-predictor-556ff547f5-4ms9v |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-bd7db-predictor-556ff547f5-4ms9v |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 20.671s (20.671s including waiting). Image size: 1306229499 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-bd7db-predictor-556ff547f5-4ms9v |
Created |
Created container: kserve-container | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-bd7db-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-bd7db-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-bd7db-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-bd7db-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x7) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-bd7db-predictor-556ff547f5-4ms9v |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.26:8080: connect: connection refused |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-bd7db |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-bd7db |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-bd7db] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-bd7db |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-bd7db |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-bd7db] is Ready | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-bd7db-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-bd7db-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-bd7db-78d659848c-xmbs5 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-raw-bd7db-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-bd7db-78d659848c |
SuccessfulCreate |
Created pod: model-chainer-raw-bd7db-78d659848c-xmbs5 | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-bd7db |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-bd7db-78d659848c from 0 to 1 | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-bd7db |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-bd7db] is Ready | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-bd7db-78d659848c-xmbs5 |
AddedInterface |
Add eth0 [10.133.0.33/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-bd7db-78d659848c-xmbs5 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-router@sha256:a4f7bd996e5317ce37dda0cef29ec36fa9a5ab15194a986a81c996dc01b5a316" | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-bd7db-78d659848c-xmbs5 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:a4f7bd996e5317ce37dda0cef29ec36fa9a5ab15194a986a81c996dc01b5a316" in 3.763s (3.763s including waiting). Image size: 216228434 bytes. | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-bd7db-78d659848c-xmbs5 |
Created |
Created container: model-chainer-raw-bd7db | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-bd7db-78d659848c-xmbs5 |
Started |
Started container model-chainer-raw-bd7db | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-bd7db-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-bd7db-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-hpa-1d91c-predictor-76b8b86b6d |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-hpa-1d91c-predictor-76b8b86b6d-vpcm5 | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-hpa-1d91c-predictor-779d987f87-tntcw |
AddedInterface |
Add eth0 [10.132.0.28/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-hpa-1d91c-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-hpa-1d91c-predictor-76b8b86b6d from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-hpa-1d91c-predictor-76b8b86b6d-vpcm5 |
AddedInterface |
Add eth0 [10.132.0.27/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-bd7db-78d659848c-xmbs5 |
Killing |
Stopping container model-chainer-raw-bd7db | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-hpa-1d91c-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-hpa-1d91c-predictor-779d987f87 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-hpa-1d91c-predictor-779d987f87 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-hpa-1d91c-predictor-779d987f87-tntcw | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-bd7db-predictor-fd55b8664-sg94q |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-1d91c-predictor-76b8b86b6d-vpcm5 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-1d91c-predictor-76b8b86b6d-vpcm5 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-1d91c-predictor-76b8b86b6d-vpcm5 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:16546eaa530f8a9a22c60a7ffa82a496f33cd9dfcf4989c8baef0968e8392589" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-1d91c-predictor-779d987f87-tntcw |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:16546eaa530f8a9a22c60a7ffa82a496f33cd9dfcf4989c8baef0968e8392589" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-bd7db-predictor-556ff547f5-4ms9v |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-1d91c-predictor-779d987f87-tntcw |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-1d91c-predictor-779d987f87-tntcw |
Started |
Started container storage-initializer | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-bd7db-predictor-fd55b8664-sg94q |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.25:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-1d91c-predictor-76b8b86b6d-vpcm5 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-1d91c-predictor-76b8b86b6d-vpcm5 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-1d91c-predictor-779d987f87-tntcw |
Pulled |
Container image "kserve/xgbserver:latest" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-1d91c-predictor-779d987f87-tntcw |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-1d91c-predictor-76b8b86b6d-vpcm5 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1362" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-1d91c-predictor-779d987f87-tntcw |
Started |
Started container kserve-container | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-1d91c-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-1d91c-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x7) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-bd7db-78d659848c-xmbs5 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-1d91c-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-1d91c-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x7) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-1d91c-predictor-779d987f87-tntcw |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.28:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-1d91c |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-1d91c |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-hpa-1d91c] is Ready | |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-1d91c-predictor-76b8b86b6d-vpcm5 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.27:8080: connect: connection refused |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-1d91c |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-1d91c |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-hpa-1d91c] is Ready | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz |
AddedInterface |
Add eth0 [10.133.0.34/23] from ovn-kubernetes | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-1d91c-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-hpa-1d91c-77dfd7db99 |
SuccessfulCreate |
Created pod: model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-1d91c-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-1d91c |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-hpa-1d91c] is Ready | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz |
Started |
Started container model-chainer-raw-hpa-1d91c | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-1d91c-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz |
Created |
Created container: model-chainer-raw-hpa-1d91c | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:a4f7bd996e5317ce37dda0cef29ec36fa9a5ab15194a986a81c996dc01b5a316" already present on machine | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-1d91c-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-hpa-1d91c |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-hpa-1d91c-77dfd7db99 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-1d91c-predictor-76b8b86b6d-vpcm5 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-1d91c-predictor-779d987f87-tntcw |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz |
Killing |
Stopping container model-chainer-raw-hpa-1d91c | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-7cc0c |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-7cc0c": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-7cc0c": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549 |
AddedInterface |
Add eth0 [10.133.0.35/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-7cc0c-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-7cc0c-predictor-7f8fd4fc94 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-7cc0c-predictor-7f8fd4fc94 |
SuccessfulCreate |
Created pod: message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549 | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549 |
Pulling |
Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549 |
Pulled |
Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.102s (1.102s including waiting). Image size: 14813193 bytes. | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-7cc0c |
InferenceServiceReady |
InferenceService [message-dumper-raw-7cc0c] is Ready | |
| (x7) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-7cc0c |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-7cc0c |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-7cc0c": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-7cc0c": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-7cc0c-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-7cc0c-predictor-7b494555d9 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-7cc0c-predictor-7b494555d9 |
SuccessfulCreate |
Created pod: isvc-logger-raw-7cc0c-predictor-7b494555d9-95ltg | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7cc0c-predictor-7b494555d9-95ltg |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-7cc0c-predictor-7b494555d9-95ltg |
AddedInterface |
Add eth0 [10.132.0.29/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7cc0c-predictor-7b494555d9-95ltg |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7cc0c-predictor-7b494555d9-95ltg |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:16546eaa530f8a9a22c60a7ffa82a496f33cd9dfcf4989c8baef0968e8392589" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7cc0c-predictor-7b494555d9-95ltg |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:cda273db4b40777305d78f4b7f8bde558e921e8f875775edae595cda2e7cfa4b" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7cc0c-predictor-7b494555d9-95ltg |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1362" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7cc0c-predictor-7b494555d9-95ltg |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7cc0c-predictor-7b494555d9-95ltg |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7cc0c-predictor-7b494555d9-95ltg |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7cc0c-predictor-7b494555d9-95ltg |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-7cc0c-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-7cc0c-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-7cc0c-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-7cc0c-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-7cc0c-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-7cc0c-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-7cc0c |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-7cc0c |
InferenceServiceReady |
InferenceService [isvc-logger-raw-7cc0c] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-scale-raw-9b575-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-scale-raw-9b575-predictor-6fd5844c66 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-9b575 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-scale-raw-9b575": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scale-raw-9b575": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-scale-raw-9b575-predictor-6fd5844c66 |
SuccessfulCreate |
Created pod: isvc-sklearn-scale-raw-9b575-predictor-6fd5844c66-whqrv | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7cc0c-predictor-7b494555d9-95ltg |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7cc0c-predictor-7b494555d9-95ltg |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-9b575-predictor-6fd5844c66-whqrv |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-scale-raw-9b575-predictor-6fd5844c66-whqrv |
AddedInterface |
Add eth0 [10.132.0.30/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-9b575-predictor-6fd5844c66-whqrv |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:16546eaa530f8a9a22c60a7ffa82a496f33cd9dfcf4989c8baef0968e8392589" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-9b575-predictor-6fd5844c66-whqrv |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-9b575-predictor-6fd5844c66-whqrv |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1362" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-9b575-predictor-6fd5844c66-whqrv |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-9b575-predictor-6fd5844c66-whqrv |
Started |
Started container kserve-container | |
| (x12) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7cc0c-predictor-7b494555d9-95ltg |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.29:8080: connect: connection refused |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-7cc0c-predictor-7b494555d9-95ltg |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-9b575-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-9b575-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-9b575 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-9b575 |
InferenceServiceReady |
InferenceService [isvc-sklearn-scale-raw-9b575] is Ready | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-0dd324 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-0dd324": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-9b575-predictor-6fd5844c66-whqrv |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
multus |
isvc-primary-0dd324-predictor-5f56f6dd6f-wvg7z |
AddedInterface |
Add eth0 [10.132.0.31/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-0dd324-predictor-5f56f6dd6f-wvg7z |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:16546eaa530f8a9a22c60a7ffa82a496f33cd9dfcf4989c8baef0968e8392589" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-0dd324-predictor-5f56f6dd6f-wvg7z |
Created |
Created container: storage-initializer | |
| (x8) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-9b575-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x8) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-9b575-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
deployment-controller |
isvc-primary-0dd324-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-primary-0dd324-predictor-5f56f6dd6f from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-primary-0dd324-predictor-5f56f6dd6f |
SuccessfulCreate |
Created pod: isvc-primary-0dd324-predictor-5f56f6dd6f-wvg7z | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-0dd324 |
UpdateFailed |
Failed to update status for InferenceService "isvc-primary-0dd324": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-0dd324": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-0dd324-predictor-5f56f6dd6f-wvg7z |
Started |
Started container storage-initializer | |
| (x16) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-9b575-predictor-6fd5844c66-whqrv |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.30:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-primary-0dd324-predictor-5f56f6dd6f-wvg7z |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-0dd324-predictor-5f56f6dd6f-wvg7z |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-0dd324-predictor-5f56f6dd6f-wvg7z |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1362" already present on machine | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-0dd324-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-0dd324-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x14) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-0dd324 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-0dd324 |
InferenceServiceReady |
InferenceService [isvc-primary-0dd324] is Ready | |
kserve-ci-e2e-test |
multus |
isvc-secondary-0dd324-predictor-66c78d4b56-8zdcw |
AddedInterface |
Add eth0 [10.132.0.32/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-0dd324 |
UpdateFailed |
Failed to update status for InferenceService "isvc-secondary-0dd324": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-0dd324": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-secondary-0dd324-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-secondary-0dd324-predictor-66c78d4b56 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-secondary-0dd324-predictor-66c78d4b56 |
SuccessfulCreate |
Created pod: isvc-secondary-0dd324-predictor-66c78d4b56-8zdcw | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-0dd324 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-0dd324": the object has been modified; please apply your changes to the latest version and try again | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-0dd324-predictor-66c78d4b56-8zdcw |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-0dd324-predictor-66c78d4b56-8zdcw |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-0dd324-predictor-66c78d4b56-8zdcw |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:16546eaa530f8a9a22c60a7ffa82a496f33cd9dfcf4989c8baef0968e8392589" already present on machine |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-0dd324-predictor-66c78d4b56-8zdcw |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-secondary-0dd324-predictor-66c78d4b56-8zdcw_kserve-ci-e2e-test(4e49926c-52a9-466a-8bb0-4d8e3872cf1c) | |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-0dd324 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-0dd324-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-0dd324-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-0dd324-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-0dd324-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
multus |
isvc-init-fail-9f5836-predictor-585f8bcf78-z24bn |
AddedInterface |
Add eth0 [10.132.0.33/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-0dd324-predictor-5f56f6dd6f-wvg7z |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-init-fail-9f5836-predictor-585f8bcf78 |
SuccessfulCreate |
Created pod: isvc-init-fail-9f5836-predictor-585f8bcf78-z24bn | |
kserve-ci-e2e-test |
deployment-controller |
isvc-init-fail-9f5836-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-init-fail-9f5836-predictor-585f8bcf78 from 0 to 1 | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-9f5836 |
UpdateFailed |
Failed to update status for InferenceService "isvc-init-fail-9f5836": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-9f5836": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-9f5836 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-9f5836": the object has been modified; please apply your changes to the latest version and try again |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-primary-0dd324-predictor-5f56f6dd6f-wvg7z |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.31:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-9f5836-predictor-585f8bcf78-z24bn |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-9f5836-predictor-585f8bcf78-z24bn |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-9f5836-predictor-585f8bcf78-z24bn |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:16546eaa530f8a9a22c60a7ffa82a496f33cd9dfcf4989c8baef0968e8392589" already present on machine |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-9f5836 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-eca71-predictor-df48fb76 |
SuccessfulCreate |
Created pod: raw-sklearn-eca71-predictor-df48fb76-48pcr | |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-9f5836-predictor-585f8bcf78-z24bn |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-init-fail-9f5836-predictor-585f8bcf78-z24bn_kserve-ci-e2e-test(ea895d81-0422-4054-9fb6-b364dfc42f26) | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-eca71 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-eca71": the object has been modified; please apply your changes to the latest version and try again | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-eca71 |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-eca71": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-eca71": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-eca71-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-eca71-predictor-df48fb76 from 0 to 1 | |
kserve-ci-e2e-test |
multus |
raw-sklearn-eca71-predictor-df48fb76-48pcr |
AddedInterface |
Add eth0 [10.132.0.34/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-eca71-predictor-df48fb76-48pcr |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:16546eaa530f8a9a22c60a7ffa82a496f33cd9dfcf4989c8baef0968e8392589" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-eca71-predictor-df48fb76-48pcr |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-eca71-predictor-df48fb76-48pcr |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-eca71-predictor-df48fb76-48pcr |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-eca71-predictor-df48fb76-48pcr |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-eca71-predictor-df48fb76-48pcr |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1362" already present on machine | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-eca71-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-eca71-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-eca71-predictor-df48fb76-48pcr |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.34:8080: connect: connection refused |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-eca71-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-eca71-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-eca71 |
InferenceServiceReady |
InferenceService [raw-sklearn-eca71] is Ready | |
| (x14) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-eca71 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-runtime-16ff5-predictor-d968b69d4 |
SuccessfulCreate |
Created pod: raw-sklearn-runtime-16ff5-predictor-d968b69d4-q7889 | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-eca71-predictor-df48fb76-48pcr |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-16ff5 |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-runtime-16ff5": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-16ff5": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-runtime-16ff5-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-runtime-16ff5-predictor-d968b69d4 from 0 to 1 | |
kserve-ci-e2e-test |
multus |
raw-sklearn-runtime-16ff5-predictor-d968b69d4-q7889 |
AddedInterface |
Add eth0 [10.132.0.35/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-16ff5-predictor-d968b69d4-q7889 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:16546eaa530f8a9a22c60a7ffa82a496f33cd9dfcf4989c8baef0968e8392589" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-16ff5-predictor-d968b69d4-q7889 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-16ff5-predictor-d968b69d4-q7889 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-16ff5-predictor-d968b69d4-q7889 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1362" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-16ff5-predictor-d968b69d4-q7889 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-16ff5-predictor-d968b69d4-q7889 |
Created |
Created container: kserve-container | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-16ff5-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-16ff5-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x8) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-16ff5-predictor-d968b69d4-q7889 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.35:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-16ff5-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-16ff5-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-16ff5 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-16ff5 |
InferenceServiceReady |
InferenceService [raw-sklearn-runtime-16ff5] is Ready | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-16ff5-predictor-d968b69d4-q7889 |
Killing |
Stopping container kserve-container |