| Time | Namespace | Component | RelatedObject | Reason | Message |
|---|---|---|---|---|---|
kserve-ci-e2e-test |
isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf to ip-10-0-143-142.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf to ip-10-0-143-142.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-f062e-predictor-797bb4b66f-2ws4r |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-f062e-predictor-797bb4b66f-2ws4r to ip-10-0-143-142.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd to ip-10-0-143-142.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r to ip-10-0-143-142.ec2.internal | ||
kserve-ci-e2e-test |
isvc-secondary-21e027-predictor-6d849b584c-p4tz8 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-secondary-21e027-predictor-6d849b584c-p4tz8 to ip-10-0-143-142.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9 to ip-10-0-143-142.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q to ip-10-0-143-142.ec2.internal | ||
kserve-ci-e2e-test |
isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9 to ip-10-0-143-142.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-hpa-079fb-7cb556b7bb-dshl5 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-hpa-079fb-7cb556b7bb-dshl5 to ip-10-0-135-252.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4 to ip-10-0-143-142.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-02e34-699f4f88fd-9twms |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-02e34-699f4f88fd-9twms to ip-10-0-135-252.ec2.internal | ||
kserve-ci-e2e-test |
isvc-primary-21e027-predictor-59449cf66d-qpdn5 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-primary-21e027-predictor-59449cf66d-qpdn5 to ip-10-0-143-142.ec2.internal | ||
kserve-ci-e2e-test |
isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj to ip-10-0-143-142.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m to ip-10-0-143-142.ec2.internal | ||
kserve-ci-e2e-test |
deployment-controller |
isvc-raw-sklearn-batcher-fc881-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c |
SuccessfulCreate |
Created pod: isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9 | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9 |
AddedInterface |
Add eth0 [10.132.0.23/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-fc881 |
UpdateFailed |
Failed to update status for InferenceService "isvc-raw-sklearn-batcher-fc881": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-fc881": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-fc881 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-fc881": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9 |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1714" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9 |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9 |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1714" in 33.582s (33.582s including waiting). Image size: 1555846059 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9 |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.402s (2.402s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-agent@sha256:2e623de9a93ccce79950e917affe03d05d0a7eff69143d7481fdb39990c764f1" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:2e623de9a93ccce79950e917affe03d05d0a7eff69143d7481fdb39990c764f1" in 2.418s (2.418s including waiting). Image size: 239997026 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9 |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9 |
Created |
Created container: agent | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-fc881-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-fc881-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-fc881 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-fc881 |
InferenceServiceReady |
InferenceService [isvc-raw-sklearn-batcher-fc881] is Ready | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-fc881-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-fc881-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9 |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd |
AddedInterface |
Add eth0 [10.132.0.24/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m |
AddedInterface |
Add eth0 [10.132.0.25/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-02e34-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-02e34-predictor-78ff766465 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-02e34-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-02e34-predictor-78ff766465 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1714" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
| (x3) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.23:8643/healthz": dial tcp 10.132.0.23:8643: connect: connection refused |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.23:8080: connect: connection refused |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fc881-predictor-78dc4575c-49cd9 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 18.712s (18.712s including waiting). Image size: 1305579259 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m |
Started |
Started container kube-rbac-proxy | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-02e34-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-02e34-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-02e34-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-02e34-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-02e34 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-02e34] is Ready | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-02e34 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-02e34 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-02e34 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-02e34] is Ready | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-02e34-699f4f88fd-9twms |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-raw-02e34-serving-cert" not found | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-02e34 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-02e34-699f4f88fd from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-02e34-699f4f88fd |
SuccessfulCreate |
Created pod: model-chainer-raw-02e34-699f4f88fd-9twms | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-02e34 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-02e34] is Ready | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-02e34-699f4f88fd-9twms |
Pulling |
Pulling image "quay.io/opendatahub/kserve-router@sha256:f94a36664a6553b816dc04c45e472a62cdcdb731a5f667f29ab4b59a2331a34e" | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-02e34-699f4f88fd-9twms |
AddedInterface |
Add eth0 [10.134.0.32/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-02e34-699f4f88fd-9twms |
Started |
Started container model-chainer-raw-02e34 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-02e34-699f4f88fd-9twms |
Created |
Created container: model-chainer-raw-02e34 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-02e34-699f4f88fd-9twms |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:f94a36664a6553b816dc04c45e472a62cdcdb731a5f667f29ab4b59a2331a34e" in 2.303s (2.303s including waiting). Image size: 218072678 bytes. | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-02e34-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-02e34-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-02e34-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-02e34-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-02e34-699f4f88fd-9twms |
Killing |
Stopping container model-chainer-raw-02e34 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-079fb |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-hpa-079fb": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-hpa-079fb": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-hpa-079fb-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-hpa-079fb-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9 from 0 to 1 | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.24:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf |
AddedInterface |
Add eth0 [10.132.0.26/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-graph-raw-hpa-079fb-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.25:8643/healthz": dial tcp 10.132.0.25:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-02e34-predictor-78ff766465-b5kkd |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.24:8643/healthz": dial tcp 10.132.0.24:8643: connect: connection refused | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q |
AddedInterface |
Add eth0 [10.132.0.27/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q |
Created |
Created container: storage-initializer | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-02e34-predictor-59c4699bdf-58k8m |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.25:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1714" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q |
Pulled |
Container image "kserve/xgbserver:latest" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q |
Started |
Started container kube-rbac-proxy | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-02e34-699f4f88fd-9twms |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-079fb-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-02e34-699f4f88fd-9twms |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.32:8080/readyz": dial tcp 10.134.0.32:8080: connect: connection refused | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-079fb-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-079fb-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-079fb-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.26:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-079fb |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-079fb |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-hpa-079fb] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-079fb |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-079fb |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-hpa-079fb] is Ready | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-079fb-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-079fb |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-hpa-079fb] is Ready | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-079fb-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-079fb-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-hpa-079fb-7cb556b7bb |
SuccessfulCreate |
Created pod: model-chainer-raw-hpa-079fb-7cb556b7bb-dshl5 | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-hpa-079fb |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-hpa-079fb-7cb556b7bb from 0 to 1 | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-079fb-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-079fb-7cb556b7bb-dshl5 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-raw-hpa-079fb-serving-cert" not found |
kserve-ci-e2e-test |
multus |
model-chainer-raw-hpa-079fb-7cb556b7bb-dshl5 |
AddedInterface |
Add eth0 [10.134.0.33/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-079fb-7cb556b7bb-dshl5 |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:f94a36664a6553b816dc04c45e472a62cdcdb731a5f667f29ab4b59a2331a34e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-079fb-7cb556b7bb-dshl5 |
Created |
Created container: model-chainer-raw-hpa-079fb | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-079fb-7cb556b7bb-dshl5 |
Started |
Started container model-chainer-raw-hpa-079fb | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-079fb-7cb556b7bb-dshl5 |
Killing |
Stopping container model-chainer-raw-hpa-079fb | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r |
AddedInterface |
Add eth0 [10.132.0.28/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r |
Pulling |
Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-1ffb5-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-1ffb5-predictor-69db84bc5d from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-1ffb5 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-1ffb5": the object has been modified; please apply your changes to the latest version and try again | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-1ffb5 |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-1ffb5": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-1ffb5": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-1ffb5-predictor-69db84bc5d |
SuccessfulCreate |
Created pod: message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r |
Pulled |
Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.357s (1.358s including waiting). Image size: 14813193 bytes. | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.27:8643/healthz": dial tcp 10.132.0.27:8643: connect: connection refused | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-079fb-predictor-5975dc47d9-kfj7q |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.27:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-079fb-predictor-74f5878849-8f7bf |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.26:8643/healthz": dial tcp 10.132.0.26:8643: connect: connection refused | |
| (x7) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-1ffb5 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-1ffb5 |
InferenceServiceReady |
InferenceService [message-dumper-raw-1ffb5] is Ready | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-1ffb5 |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-1ffb5": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-1ffb5": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-logger-raw-1ffb5-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-1ffb5-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-1ffb5-predictor-fb65b5494 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-1ffb5-predictor-fb65b5494 |
SuccessfulCreate |
Created pod: isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf |
AddedInterface |
Add eth0 [10.132.0.29/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1714" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:2e623de9a93ccce79950e917affe03d05d0a7eff69143d7481fdb39990c764f1" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf |
Started |
Started container kube-rbac-proxy | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-079fb-7cb556b7bb-dshl5 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-1ffb5-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-1ffb5-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-1ffb5-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-1ffb5-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-1ffb5 |
InferenceServiceReady |
InferenceService [isvc-logger-raw-1ffb5] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-1ffb5 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-1ffb5-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-1ffb5-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-3cac6 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-scale-raw-3cac6": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scale-raw-3cac6": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.28:8643/healthz": dial tcp 10.132.0.28:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-1ffb5-predictor-69db84bc5d-k2k5r |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-scale-raw-3cac6-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-scale-raw-3cac6-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc |
SuccessfulCreate |
Created pod: isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9 | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.29:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9 |
AddedInterface |
Add eth0 [10.132.0.30/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1714" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9 |
Started |
Started container kserve-container | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-1ffb5-predictor-fb65b5494-k2vgf |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.29:8643/healthz": dial tcp 10.132.0.29:8643: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-3cac6-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-3cac6-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x8) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-3cac6-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x8) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-3cac6-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-3cac6 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-3cac6 |
InferenceServiceReady |
InferenceService [isvc-sklearn-scale-raw-3cac6] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9 |
Killing |
Stopping container kserve-container | |
| (x18) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.30:8080: connect: connection refused |
kserve-ci-e2e-test |
replicaset-controller |
isvc-primary-21e027-predictor-59449cf66d |
SuccessfulCreate |
Created pod: isvc-primary-21e027-predictor-59449cf66d-qpdn5 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-21e027 |
UpdateFailed |
Failed to update status for InferenceService "isvc-primary-21e027": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-21e027": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-21e027 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-21e027": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-primary-21e027-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-primary-21e027-predictor-59449cf66d from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-21e027-predictor-59449cf66d-qpdn5 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-21e027-predictor-59449cf66d-qpdn5 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-primary-21e027-predictor-59449cf66d-qpdn5 |
AddedInterface |
Add eth0 [10.132.0.31/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-21e027-predictor-59449cf66d-qpdn5 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-3cac6-predictor-85cd6d56dc-lv8p9 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.30:8643/healthz": dial tcp 10.132.0.30:8643: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-primary-21e027-predictor-59449cf66d-qpdn5 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-21e027-predictor-59449cf66d-qpdn5 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-21e027-predictor-59449cf66d-qpdn5 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-21e027-predictor-59449cf66d-qpdn5 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-21e027-predictor-59449cf66d-qpdn5 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-21e027-predictor-59449cf66d-qpdn5 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1714" already present on machine | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-21e027-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-21e027-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-primary-21e027-predictor-59449cf66d-qpdn5 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.31:8080: connect: connection refused |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-21e027 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-21e027 |
InferenceServiceReady |
InferenceService [isvc-primary-21e027] is Ready | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-secondary-21e027-predictor-6d849b584c |
SuccessfulCreate |
Created pod: isvc-secondary-21e027-predictor-6d849b584c-p4tz8 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-secondary-21e027-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-secondary-21e027-predictor-6d849b584c from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-21e027 |
UpdateFailed |
Failed to update status for InferenceService "isvc-secondary-21e027": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-21e027": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-21e027 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-21e027": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-21e027-predictor-6d849b584c-p4tz8 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-secondary-21e027-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
multus |
isvc-secondary-21e027-predictor-6d849b584c-p4tz8 |
AddedInterface |
Add eth0 [10.132.0.32/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-21e027-predictor-6d849b584c-p4tz8 |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-21e027-predictor-6d849b584c-p4tz8 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-21e027-predictor-6d849b584c-p4tz8 |
Created |
Created container: storage-initializer |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-21e027-predictor-6d849b584c-p4tz8 |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-secondary-21e027-predictor-6d849b584c-p4tz8_kserve-ci-e2e-test(d3536bec-6d76-4d3e-ae0c-9d38806500f8) | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-21e027 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-21e027-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-21e027-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-21e027-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-21e027-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-21e027-predictor-59449cf66d-qpdn5 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-21e027-predictor-59449cf66d-qpdn5 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-afe32a |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-afe32a": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-afe32a |
UpdateFailed |
Failed to update status for InferenceService "isvc-init-fail-afe32a": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-afe32a": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-init-fail-afe32a-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-init-fail-afe32a-predictor-7f95c54965 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-init-fail-afe32a-predictor-7f95c54965 |
SuccessfulCreate |
Created pod: isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj | |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-init-fail-afe32a-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
multus |
isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj |
AddedInterface |
Add eth0 [10.132.0.33/23] from ovn-kubernetes | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-afe32a |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-afe32a-predictor-7f95c54965-n9tnj |
Killing |
Stopping container storage-initializer | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-f062e-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-f062e-predictor-797bb4b66f from 0 to 1 | |
kserve-ci-e2e-test |
multus |
raw-sklearn-f062e-predictor-797bb4b66f-2ws4r |
AddedInterface |
Add eth0 [10.132.0.34/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-f062e-predictor-797bb4b66f-2ws4r |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-f062e-predictor-797bb4b66f |
SuccessfulCreate |
Created pod: raw-sklearn-f062e-predictor-797bb4b66f-2ws4r | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-f062e-predictor-797bb4b66f-2ws4r |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-f062e-predictor-797bb4b66f-2ws4r |
Created |
Created container: storage-initializer | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-f062e |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-f062e": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-f062e": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-f062e |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-f062e": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-f062e-predictor-797bb4b66f-2ws4r |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-f062e-predictor-797bb4b66f-2ws4r |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-f062e-predictor-797bb4b66f-2ws4r |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-f062e-predictor-797bb4b66f-2ws4r |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-f062e-predictor-797bb4b66f-2ws4r |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-f062e-predictor-797bb4b66f-2ws4r |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1714" already present on machine | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-f062e-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-f062e-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-f062e |
InferenceServiceReady |
InferenceService [raw-sklearn-f062e] is Ready | |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-f062e |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-f062e-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-f062e-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-f062e-predictor-797bb4b66f-2ws4r |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-f062e-predictor-797bb4b66f-2ws4r |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-f062e-predictor-797bb4b66f-2ws4r |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.34:8643/healthz": dial tcp 10.132.0.34:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-f062e-predictor-797bb4b66f-2ws4r |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.34:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-4d85b |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-runtime-4d85b": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-4d85b": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-runtime-4d85b-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-runtime-4d85b-predictor-5846b9b6b9 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-runtime-4d85b-predictor-5846b9b6b9 |
SuccessfulCreate |
Created pod: raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4 | |
kserve-ci-e2e-test |
multus |
raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4 |
AddedInterface |
Add eth0 [10.132.0.35/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1714" already present on machine | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-4d85b-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-4d85b-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-4d85b |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-4d85b |
InferenceServiceReady |
InferenceService [raw-sklearn-runtime-4d85b] is Ready | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4 |
Killing |
Stopping container kserve-container | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-4d85b-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-4d85b-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.35:8643/healthz": dial tcp 10.132.0.35:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-4d85b-predictor-5846b9b6b9-mm9h4 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.35:8080: connect: connection refused |