| Time | Namespace | Component | RelatedObject | Reason | Message |
|---|---|---|---|---|---|
kserve-ci-e2e-test |
model-chainer-raw-b4160-658c8f4f6-gtscw |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-b4160-658c8f4f6-gtscw to ip-10-0-129-46.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m to ip-10-0-139-250.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-ef99f-predictor-77b85567df-trpsw |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-ef99f-predictor-77b85567df-trpsw to ip-10-0-139-250.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-hpa-88d19-66946dd7d5-47rc7 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-hpa-88d19-66946dd7d5-47rc7 to ip-10-0-129-46.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn to ip-10-0-139-250.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg to ip-10-0-139-250.ec2.internal | ||
kserve-ci-e2e-test |
isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2 to ip-10-0-139-250.ec2.internal | ||
kserve-ci-e2e-test |
isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql to ip-10-0-139-250.ec2.internal | ||
kserve-ci-e2e-test |
isvc-primary-f8d41a-predictor-656db64bb8-495pz |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-primary-f8d41a-predictor-656db64bb8-495pz to ip-10-0-139-250.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs to ip-10-0-139-250.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx to ip-10-0-139-250.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw to ip-10-0-139-250.ec2.internal | ||
kserve-ci-e2e-test |
isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6 to ip-10-0-139-250.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj to ip-10-0-139-250.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-e7cb2-predictor-689764df45-x5zcv |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-e7cb2-predictor-689764df45-x5zcv to ip-10-0-139-250.ec2.internal | ||
kserve-ci-e2e-test |
deployment-controller |
isvc-raw-sklearn-batcher-ea0d4-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-ea0d4 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-ea0d4": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql |
AddedInterface |
Add eth0 [10.134.0.20/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5 |
SuccessfulCreate |
Created pod: isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-ea0d4 |
UpdateFailed |
Failed to update status for InferenceService "isvc-raw-sklearn-batcher-ea0d4": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-ea0d4": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" in 4.498s (4.498s including waiting). Image size: 300633537 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1703" | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-ea0d4-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-ea0d4-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1703" in 16.609s (16.609s including waiting). Image size: 1555746730 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql |
Pulling |
Pulling image "quay.io/opendatahub/kserve-agent@sha256:0706c86054be2f22ee66c8739e97daebd0094b98a978f91cd3fb4f59862bad3f" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.748s (2.748s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:0706c86054be2f22ee66c8739e97daebd0094b98a978f91cd3fb4f59862bad3f" in 3.306s (3.306s including waiting). Image size: 239997029 bytes. | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-ea0d4-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1/namespaces/kserve-ci-e2e-test/pods?labelSelector=app%3Disvc.isvc-raw-sklearn-batcher-ea0d4-predictor\": Post \"https://172.31.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)") has prevented the request from succeeding (get pods.metrics.k8s.io) |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-ea0d4-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1/namespaces/kserve-ci-e2e-test/pods?labelSelector=app%3Disvc.isvc-raw-sklearn-batcher-ea0d4-predictor\": Post \"https://172.31.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)") has prevented the request from succeeding (get pods.metrics.k8s.io) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-ea0d4-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-ea0d4-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-ea0d4 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-ea0d4 |
InferenceServiceReady |
InferenceService [isvc-raw-sklearn-batcher-ea0d4] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-b4160-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-b4160-predictor-795b89786f |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-b4160-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-b4160-predictor-795b89786f from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-b4160 |
UpdateFailed |
Failed to update status for InferenceService "isvc-xgboost-graph-raw-b4160": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-graph-raw-b4160": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-b4160 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-b4160": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-b4160": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs |
AddedInterface |
Add eth0 [10.134.0.22/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj |
AddedInterface |
Add eth0 [10.134.0.21/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1703" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.20:8080: connect: connection refused |
| (x3) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-ea0d4-predictor-f8dcfb6b5-tvxql |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.20:8643/healthz": dial tcp 10.134.0.20:8643: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 17.481s (17.481s including waiting). Image size: 1305293053 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs |
Created |
Created container: kserve-container | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-b4160-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-b4160-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-b4160-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-b4160-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.22:8080: connect: connection refused |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-b4160 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-b4160 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-b4160] is Ready | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-b4160 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-b4160] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-b4160 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-b4160-658c8f4f6 |
SuccessfulCreate |
Created pod: model-chainer-raw-b4160-658c8f4f6-gtscw | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-b4160 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-b4160-658c8f4f6 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-b4160-658c8f4f6-gtscw |
Pulling |
Pulling image "quay.io/opendatahub/kserve-router@sha256:a8d64b572a976cc85b28488a3ef8f7d50ce28ddb75186c1859612e88d6dba080" | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-b4160-658c8f4f6-gtscw |
AddedInterface |
Add eth0 [10.133.0.26/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-b4160 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-b4160] is Ready | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-b4160-658c8f4f6-gtscw |
Created |
Created container: model-chainer-raw-b4160 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-b4160-658c8f4f6-gtscw |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:a8d64b572a976cc85b28488a3ef8f7d50ce28ddb75186c1859612e88d6dba080" in 2.057s (2.057s including waiting). Image size: 218071146 bytes. | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-b4160-658c8f4f6-gtscw |
Started |
Started container model-chainer-raw-b4160 | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-b4160-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-b4160-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-b4160-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-b4160-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-b4160-658c8f4f6-gtscw |
Killing |
Stopping container model-chainer-raw-b4160 | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn |
AddedInterface |
Add eth0 [10.134.0.23/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-hpa-88d19-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-88d19 |
UpdateFailed |
Failed to update status for InferenceService "isvc-xgboost-graph-raw-hpa-88d19": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-graph-raw-hpa-88d19": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-hpa-88d19-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-88d19 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-hpa-88d19": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-hpa-88d19": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.21:8643/healthz": dial tcp 10.134.0.21:8643: connect: connection refused | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx |
AddedInterface |
Add eth0 [10.134.0.24/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx |
Created |
Created container: storage-initializer | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-b4160-predictor-bdd664fb5-4wtfj |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.21:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-b4160-predictor-795b89786f-nlmbs |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.22:8643/healthz": dial tcp 10.134.0.22:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx |
Pulled |
Container image "kserve/xgbserver:latest" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1703" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn |
Created |
Created container: kube-rbac-proxy | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-b4160-658c8f4f6-gtscw |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-88d19-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-88d19-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-88d19-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-88d19-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-88d19 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-hpa-88d19] is Ready | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-88d19 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-88d19 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-hpa-88d19] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-88d19 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-88d19-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-88d19-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-88d19 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-hpa-88d19] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-hpa-88d19 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-hpa-88d19-66946dd7d5 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-hpa-88d19-66946dd7d5 |
SuccessfulCreate |
Created pod: model-chainer-raw-hpa-88d19-66946dd7d5-47rc7 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-88d19-66946dd7d5-47rc7 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-raw-hpa-88d19-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-88d19-66946dd7d5-47rc7 |
Created |
Created container: model-chainer-raw-hpa-88d19 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-88d19-66946dd7d5-47rc7 |
Started |
Started container model-chainer-raw-hpa-88d19 | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-hpa-88d19-66946dd7d5-47rc7 |
AddedInterface |
Add eth0 [10.133.0.27/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-88d19-66946dd7d5-47rc7 |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:a8d64b572a976cc85b28488a3ef8f7d50ce28ddb75186c1859612e88d6dba080" already present on machine | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-88d19-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-88d19-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-e7cb2 |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-e7cb2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-e7cb2": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-e7cb2-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-e7cb2-predictor-689764df45 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-88d19-66946dd7d5-47rc7 |
Killing |
Stopping container model-chainer-raw-hpa-88d19 | |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-e7cb2-predictor-689764df45 |
SuccessfulCreate |
Created pod: message-dumper-raw-e7cb2-predictor-689764df45-x5zcv | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-e7cb2-predictor-689764df45-x5zcv |
Pulling |
Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-e7cb2-predictor-689764df45-x5zcv |
AddedInterface |
Add eth0 [10.134.0.25/23] from ovn-kubernetes | |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.24:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-e7cb2-predictor-689764df45-x5zcv |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.23:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-88d19-predictor-7b9b479bb5-x6hzx |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.24:8643/healthz": dial tcp 10.134.0.24:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-e7cb2-predictor-689764df45-x5zcv |
Pulled |
Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.325s (1.325s including waiting). Image size: 14813193 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-88d19-predictor-86cbcdf9c-276xn |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.23:8643/healthz": dial tcp 10.134.0.23:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-e7cb2-predictor-689764df45-x5zcv |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-e7cb2-predictor-689764df45-x5zcv |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-e7cb2-predictor-689764df45-x5zcv |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-e7cb2-predictor-689764df45-x5zcv |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-e7cb2 |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-e7cb2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-e7cb2": the object has been modified; please apply your changes to the latest version and try again | |
| (x7) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-e7cb2 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-e7cb2-predictor-6748c49bb5 |
SuccessfulCreate |
Created pod: isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-e7cb2 |
InferenceServiceReady |
InferenceService [message-dumper-raw-e7cb2] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-e7cb2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-e7cb2-predictor-6748c49bb5 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw |
AddedInterface |
Add eth0 [10.134.0.26/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:0706c86054be2f22ee66c8739e97daebd0094b98a978f91cd3fb4f59862bad3f" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1703" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw |
Created |
Created container: kube-rbac-proxy | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-88d19-66946dd7d5-47rc7 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-e7cb2-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-e7cb2-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-e7cb2-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-e7cb2-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-e7cb2 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-e7cb2 |
InferenceServiceReady |
InferenceService [isvc-logger-raw-e7cb2] is Ready | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-e7cb2-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-e7cb2-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw |
Killing |
Stopping container kube-rbac-proxy | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-scale-raw-26dd6-predictor-serving-cert" not found |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-26dd6 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-scale-raw-26dd6": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scale-raw-26dd6": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-e7cb2-predictor-689764df45-x5zcv |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-e7cb2-predictor-689764df45-x5zcv |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-scale-raw-26dd6-predictor-7468f89667 |
SuccessfulCreate |
Created pod: isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-scale-raw-26dd6-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-scale-raw-26dd6-predictor-7468f89667 from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg |
AddedInterface |
Add eth0 [10.134.0.27/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1703" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg |
Started |
Started container kube-rbac-proxy | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.26:8080: connect: connection refused |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x4) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-e7cb2-predictor-6748c49bb5-p7bgw |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.26:8643/healthz": dial tcp 10.134.0.26:8643: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-26dd6-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-26dd6-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-26dd6 |
InferenceServiceReady |
InferenceService [isvc-sklearn-scale-raw-26dd6] is Ready | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-26dd6 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x9) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-26dd6-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x9) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-26dd6-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x18) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.27:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-f8d41a-predictor-656db64bb8-495pz |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-f8d41a-predictor-656db64bb8-495pz |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-f8d41a |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-f8d41a": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-f8d41a |
UpdateFailed |
Failed to update status for InferenceService "isvc-primary-f8d41a": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-f8d41a": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-primary-f8d41a-predictor-656db64bb8-495pz |
AddedInterface |
Add eth0 [10.134.0.28/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-primary-f8d41a-predictor-656db64bb8 |
SuccessfulCreate |
Created pod: isvc-primary-f8d41a-predictor-656db64bb8-495pz | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-f8d41a-predictor-656db64bb8-495pz |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine | |
kserve-ci-e2e-test |
deployment-controller |
isvc-primary-f8d41a-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-primary-f8d41a-predictor-656db64bb8 from 0 to 1 | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-26dd6-predictor-7468f89667-b9zdg |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.27:8643/healthz": dial tcp 10.134.0.27:8643: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-primary-f8d41a-predictor-656db64bb8-495pz |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-f8d41a-predictor-656db64bb8-495pz |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-f8d41a-predictor-656db64bb8-495pz |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1703" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-f8d41a-predictor-656db64bb8-495pz |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-f8d41a-predictor-656db64bb8-495pz |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-f8d41a-predictor-656db64bb8-495pz |
Started |
Started container kube-rbac-proxy | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-f8d41a-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-f8d41a-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-primary-f8d41a-predictor-656db64bb8-495pz |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.28:8080: connect: connection refused |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-f8d41a |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-f8d41a |
InferenceServiceReady |
InferenceService [isvc-primary-f8d41a] is Ready | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-f8d41a-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-f8d41a |
UpdateFailed |
Failed to update status for InferenceService "isvc-secondary-f8d41a": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-f8d41a": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-f8d41a |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-f8d41a": the object has been modified; please apply your changes to the latest version and try again |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-f8d41a-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
deployment-controller |
isvc-secondary-f8d41a-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-secondary-f8d41a-predictor-655bb967ff from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-secondary-f8d41a-predictor-655bb967ff |
SuccessfulCreate |
Created pod: isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6 | |
kserve-ci-e2e-test |
multus |
isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6 |
AddedInterface |
Add eth0 [10.134.0.29/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6 |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6 |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6 |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-secondary-f8d41a-predictor-655bb967ff-8f9j6_kserve-ci-e2e-test(47e6f04f-745d-4ace-8dd1-477f950a7aaf) | |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-f8d41a |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-primary-f8d41a-predictor-656db64bb8-495pz |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-f8d41a-predictor-656db64bb8-495pz |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-f8d41a-predictor-656db64bb8-495pz |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.28:8643/healthz": dial tcp 10.134.0.28:8643: connect: connection refused | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-1558a9 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-1558a9": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-init-fail-1558a9-predictor-7d45464cf5 |
SuccessfulCreate |
Created pod: isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-init-fail-1558a9-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-init-fail-1558a9-predictor-7d45464cf5 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-1558a9 |
UpdateFailed |
Failed to update status for InferenceService "isvc-init-fail-1558a9": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-1558a9": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2 |
AddedInterface |
Add eth0 [10.134.0.30/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2 |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine |
| (x8) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-1558a9 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2 |
Created |
Created container: storage-initializer |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-1558a9-predictor-7d45464cf5-9p4c2 |
Killing |
Stopping container storage-initializer | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-ef99f-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-ef99f-predictor-77b85567df from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-ef99f |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-ef99f": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-ef99f": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-ef99f-predictor-77b85567df-trpsw |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
raw-sklearn-ef99f-predictor-77b85567df-trpsw |
AddedInterface |
Add eth0 [10.134.0.31/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-ef99f-predictor-77b85567df |
SuccessfulCreate |
Created pod: raw-sklearn-ef99f-predictor-77b85567df-trpsw | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-ef99f-predictor-77b85567df-trpsw |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-ef99f-predictor-77b85567df-trpsw |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-ef99f-predictor-77b85567df-trpsw |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-ef99f-predictor-77b85567df-trpsw |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-ef99f-predictor-77b85567df-trpsw |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-ef99f-predictor-77b85567df-trpsw |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-ef99f-predictor-77b85567df-trpsw |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1703" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-ef99f-predictor-77b85567df-trpsw |
Created |
Created container: kserve-container | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-ef99f-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-ef99f-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-ef99f |
InferenceServiceReady |
InferenceService [raw-sklearn-ef99f] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-ef99f |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-ef99f-predictor-77b85567df-trpsw |
Killing |
Stopping container kube-rbac-proxy | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-ef99f-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-ef99f-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-ef99f-predictor-77b85567df-trpsw |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-ef99f-predictor-77b85567df-trpsw |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.31:8643/healthz": dial tcp 10.134.0.31:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-ef99f-predictor-77b85567df-trpsw |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.31:8080: connect: connection refused |
kserve-ci-e2e-test |
multus |
raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m |
AddedInterface |
Add eth0 [10.134.0.32/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-321a7 |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-runtime-321a7": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-321a7": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-runtime-321a7-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-runtime-321a7-predictor-c6d947c4f from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-runtime-321a7-predictor-c6d947c4f |
SuccessfulCreate |
Created pod: raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1703" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m |
Created |
Created container: kserve-container | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-321a7-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-321a7-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-321a7 |
InferenceServiceReady |
InferenceService [raw-sklearn-runtime-321a7] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-321a7 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m |
Killing |
Stopping container kserve-container | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-321a7-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-321a7-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.32:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-321a7-predictor-c6d947c4f-2tl5m |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.32:8643/healthz": dial tcp 10.134.0.32:8643: connect: connection refused |