| Time | Namespace | Component | RelatedObject | Reason | Message |
|---|---|---|---|---|---|
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd to ip-10-0-137-85.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck to ip-10-0-137-85.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk to ip-10-0-137-85.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-a32f7-ff9c4f8f9-tjp8w |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-a32f7-ff9c4f8f9-tjp8w to ip-10-0-137-177.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x to ip-10-0-137-85.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr to ip-10-0-137-85.ec2.internal | ||
kserve-ci-e2e-test |
isvc-init-fail-279bfc-predictor-866d647b77-4nw6z |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-init-fail-279bfc-predictor-866d647b77-4nw6z to ip-10-0-137-85.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-hpa-7f37f-987485c9-gstnr |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-hpa-7f37f-987485c9-gstnr to ip-10-0-137-177.ec2.internal | ||
kserve-ci-e2e-test |
isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc to ip-10-0-137-85.ec2.internal | ||
kserve-ci-e2e-test |
isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s to ip-10-0-137-85.ec2.internal | ||
kserve-ci-e2e-test |
isvc-primary-9e75a1-predictor-7fd479db86-64jt7 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-primary-9e75a1-predictor-7fd479db86-64jt7 to ip-10-0-137-85.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-164a6-predictor-747b66f544-qkntm |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-164a6-predictor-747b66f544-qkntm to ip-10-0-137-85.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp to ip-10-0-137-85.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz to ip-10-0-137-85.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj to ip-10-0-137-85.ec2.internal | ||
kserve-ci-e2e-test |
deployment-controller |
isvc-raw-sklearn-batcher-5f43b-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:5d63c390841fcd6358d7b4068789845000a8eff3dfe19f0f9b697dde4d31930a" | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-5f43b |
UpdateFailed |
Failed to update status for InferenceService "isvc-raw-sklearn-batcher-5f43b": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-5f43b": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-5f43b |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-5f43b": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
replicaset-controller |
isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887 |
SuccessfulCreate |
Created pod: isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc | |
kserve-ci-e2e-test |
multus |
isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc |
AddedInterface |
Add eth0 [10.134.0.16/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:5d63c390841fcd6358d7b4068789845000a8eff3dfe19f0f9b697dde4d31930a" in 3.298s (3.298s including waiting). Image size: 300574656 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1745" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1745" in 13.03s (13.03s including waiting). Image size: 1555872170 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc |
Pulling |
Pulling image "quay.io/opendatahub/kserve-agent@sha256:c30f46bbbf240f23d63e3057afecb3d51f030f01e92352c91c1f6bf13b53e7c1" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 6.694s (6.694s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:c30f46bbbf240f23d63e3057afecb3d51f030f01e92352c91c1f6bf13b53e7c1" in 3.76s (3.76s including waiting). Image size: 240005729 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc |
Started |
Started container agent | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-5f43b-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-5f43b-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-5f43b-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-5f43b-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-5f43b |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-5f43b |
InferenceServiceReady |
InferenceService [isvc-raw-sklearn-batcher-5f43b] is Ready | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-a32f7 |
UpdateFailed |
Failed to update status for InferenceService "isvc-xgboost-graph-raw-a32f7": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-graph-raw-a32f7": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-a32f7-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-a32f7 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-a32f7": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-a32f7": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-a32f7-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-graph-raw-a32f7-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-graph-raw-a32f7-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5d63c390841fcd6358d7b4068789845000a8eff3dfe19f0f9b697dde4d31930a" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x |
AddedInterface |
Add eth0 [10.134.0.17/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz |
AddedInterface |
Add eth0 [10.134.0.18/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5d63c390841fcd6358d7b4068789845000a8eff3dfe19f0f9b697dde4d31930a" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1745" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x |
Created |
Created container: kube-rbac-proxy | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 18.414s (18.414s including waiting). Image size: 1306969338 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz |
Created |
Created container: kube-rbac-proxy | |
| (x6) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.16:8643/healthz": dial tcp 10.134.0.16:8643: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5f43b-predictor-59fd64887-s52wc |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.16:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-a32f7-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-a32f7-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-a32f7-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-a32f7-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-a32f7 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-a32f7 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-a32f7] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-a32f7 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-a32f7 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-a32f7] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-a32f7 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-a32f7-ff9c4f8f9 from 0 to 1 | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-a32f7 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-a32f7] is Ready | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-a32f7-ff9c4f8f9 |
SuccessfulCreate |
Created pod: model-chainer-raw-a32f7-ff9c4f8f9-tjp8w | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-a32f7-ff9c4f8f9-tjp8w |
AddedInterface |
Add eth0 [10.132.0.33/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-a32f7-ff9c4f8f9-tjp8w |
Pulling |
Pulling image "quay.io/opendatahub/kserve-router@sha256:7e35538adc84c05d9420d87001e9355aeddffd6326bb391010f8e6c1a3c85203" | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-a32f7-ff9c4f8f9-tjp8w |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:7e35538adc84c05d9420d87001e9355aeddffd6326bb391010f8e6c1a3c85203" in 2.578s (2.578s including waiting). Image size: 218081378 bytes. | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-a32f7-ff9c4f8f9-tjp8w |
Created |
Created container: model-chainer-raw-a32f7 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-a32f7-ff9c4f8f9-tjp8w |
Started |
Started container model-chainer-raw-a32f7 | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-a32f7-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-a32f7-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-a32f7-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-a32f7-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-hpa-7f37f-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-7f37f |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-hpa-7f37f": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-hpa-7f37f": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-a32f7-ff9c4f8f9-tjp8w |
Killing |
Stopping container model-chainer-raw-a32f7 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-hpa-7f37f-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-graph-raw-hpa-7f37f-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd |
AddedInterface |
Add eth0 [10.134.0.19/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-graph-raw-hpa-7f37f-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.18:8643/healthz": dial tcp 10.134.0.18:8643: connect: connection refused | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-a32f7-predictor-686c679c4f-btqdz |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.18:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5d63c390841fcd6358d7b4068789845000a8eff3dfe19f0f9b697dde4d31930a" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj |
AddedInterface |
Add eth0 [10.134.0.20/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5d63c390841fcd6358d7b4068789845000a8eff3dfe19f0f9b697dde4d31930a" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj |
Created |
Created container: storage-initializer | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.17:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-a32f7-predictor-c9d466f5f-pnl7x |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.17:8643/healthz": dial tcp 10.134.0.17:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1745" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj |
Pulled |
Container image "kserve/xgbserver:latest" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-7f37f-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-7f37f-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-a32f7-ff9c4f8f9-tjp8w |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-7f37f-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-7f37f-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-7f37f |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-hpa-7f37f] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-7f37f |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-7f37f |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-hpa-7f37f] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-7f37f |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-7f37f-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-7f37f-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-hpa-7f37f |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-hpa-7f37f-987485c9 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-hpa-7f37f-987485c9 |
SuccessfulCreate |
Created pod: model-chainer-raw-hpa-7f37f-987485c9-gstnr | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-7f37f |
UpdateFailed |
Failed to update status for InferenceGraph "model-chainer-raw-hpa-7f37f": Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "model-chainer-raw-hpa-7f37f": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-7f37f |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-hpa-7f37f] is Ready | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-7f37f-987485c9-gstnr |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-raw-hpa-7f37f-serving-cert" not found | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-7f37f |
InternalError |
fails to update InferenceGraph status: Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "model-chainer-raw-hpa-7f37f": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-hpa-7f37f-987485c9-gstnr |
AddedInterface |
Add eth0 [10.132.0.34/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-7f37f-987485c9-gstnr |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:7e35538adc84c05d9420d87001e9355aeddffd6326bb391010f8e6c1a3c85203" already present on machine | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-7f37f-987485c9-gstnr |
Started |
Started container model-chainer-raw-hpa-7f37f | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-7f37f-987485c9-gstnr |
Created |
Created container: model-chainer-raw-hpa-7f37f | |
| (x6) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-7f37f-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x6) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-7f37f-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-164a6-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-164a6-predictor-7cb87dc86 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-164a6-predictor-7cb87dc86 |
SuccessfulCreate |
Created pod: message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "message-dumper-raw-164a6-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-7f37f-987485c9-gstnr |
Killing |
Stopping container model-chainer-raw-hpa-7f37f | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-164a6 |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-164a6": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-164a6": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.19:8643/healthz": dial tcp 10.134.0.19:8643: connect: connection refused | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr |
AddedInterface |
Add eth0 [10.134.0.21/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr |
Pulling |
Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.20:8643/healthz": dial tcp 10.134.0.20:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-7f37f-predictor-785ff475bc-5jwpd |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.19:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr |
Pulled |
Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.361s (1.361s including waiting). Image size: 14813193 bytes. | |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-7f37f-predictor-577799f8b5-92zkj |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.20:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-164a6 |
InferenceServiceReady |
InferenceService [message-dumper-raw-164a6] is Ready | |
| (x8) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-164a6 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-164a6-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-164a6-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-164a6 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-164a6": the object has been modified; please apply your changes to the latest version and try again | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-164a6 |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-164a6": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-164a6": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-164a6-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-164a6-predictor-747b66f544 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-164a6-predictor-747b66f544 |
SuccessfulCreate |
Created pod: isvc-logger-raw-164a6-predictor-747b66f544-qkntm | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-164a6-predictor-747b66f544-qkntm |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-logger-raw-164a6-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-164a6-predictor-747b66f544-qkntm |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-164a6-predictor-747b66f544-qkntm |
AddedInterface |
Add eth0 [10.134.0.22/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-164a6-predictor-747b66f544-qkntm |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5d63c390841fcd6358d7b4068789845000a8eff3dfe19f0f9b697dde4d31930a" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-164a6-predictor-747b66f544-qkntm |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-164a6-predictor-747b66f544-qkntm |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-164a6-predictor-747b66f544-qkntm |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-164a6-predictor-747b66f544-qkntm |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-164a6-predictor-747b66f544-qkntm |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-164a6-predictor-747b66f544-qkntm |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-164a6-predictor-747b66f544-qkntm |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-164a6-predictor-747b66f544-qkntm |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1745" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-164a6-predictor-747b66f544-qkntm |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-164a6-predictor-747b66f544-qkntm |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:c30f46bbbf240f23d63e3057afecb3d51f030f01e92352c91c1f6bf13b53e7c1" already present on machine | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-7f37f-987485c9-gstnr |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-164a6-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-164a6-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-164a6-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-164a6-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-164a6 |
InferenceServiceReady |
InferenceService [isvc-logger-raw-164a6] is Ready | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-164a6 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-164a6-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-164a6-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-scale-raw-e4053-predictor-66769ddddf |
SuccessfulCreate |
Created pod: isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-e4053 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-scale-raw-e4053": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scale-raw-e4053": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-164a6-predictor-7cb87dc86-5r5rr |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-164a6-predictor-747b66f544-qkntm |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-164a6-predictor-747b66f544-qkntm |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-164a6-predictor-747b66f544-qkntm |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-scale-raw-e4053-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-scale-raw-e4053-predictor-66769ddddf from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck |
AddedInterface |
Add eth0 [10.134.0.23/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5d63c390841fcd6358d7b4068789845000a8eff3dfe19f0f9b697dde4d31930a" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1745" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck |
Started |
Started container kserve-container | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-164a6-predictor-747b66f544-qkntm |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.22:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-164a6-predictor-747b66f544-qkntm |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-e4053-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-e4053-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x4) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-164a6-predictor-747b66f544-qkntm |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.22:8643/healthz": dial tcp 10.134.0.22:8643: connect: connection refused |
| (x9) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-e4053-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x9) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-e4053-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-e4053 |
InferenceServiceReady |
InferenceService [isvc-sklearn-scale-raw-e4053] is Ready | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-e4053 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck |
Killing |
Stopping container kube-rbac-proxy | |
| (x17) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.23:8080: connect: connection refused |
kserve-ci-e2e-test |
deployment-controller |
isvc-primary-9e75a1-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-primary-9e75a1-predictor-7fd479db86 from 0 to 1 | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-9e75a1 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-9e75a1": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-primary-9e75a1-predictor-7fd479db86-64jt7 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-primary-9e75a1-predictor-7fd479db86 |
SuccessfulCreate |
Created pod: isvc-primary-9e75a1-predictor-7fd479db86-64jt7 | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-9e75a1-predictor-7fd479db86-64jt7 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5d63c390841fcd6358d7b4068789845000a8eff3dfe19f0f9b697dde4d31930a" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-9e75a1-predictor-7fd479db86-64jt7 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-primary-9e75a1-predictor-7fd479db86-64jt7 |
AddedInterface |
Add eth0 [10.134.0.24/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-9e75a1 |
UpdateFailed |
Failed to update status for InferenceService "isvc-primary-9e75a1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-9e75a1": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e4053-predictor-66769ddddf-wbsck |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.23:8643/healthz": dial tcp 10.134.0.23:8643: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-primary-9e75a1-predictor-7fd479db86-64jt7 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1745" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-9e75a1-predictor-7fd479db86-64jt7 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-9e75a1-predictor-7fd479db86-64jt7 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-9e75a1-predictor-7fd479db86-64jt7 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-9e75a1-predictor-7fd479db86-64jt7 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-9e75a1-predictor-7fd479db86-64jt7 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-9e75a1-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-9e75a1-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-9e75a1 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-9e75a1 |
InferenceServiceReady |
InferenceService [isvc-primary-9e75a1] is Ready | |
kserve-ci-e2e-test |
multus |
isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s |
AddedInterface |
Add eth0 [10.134.0.25/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-secondary-9e75a1-predictor-7b56c4bfc |
SuccessfulCreate |
Created pod: isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s | |
kserve-ci-e2e-test |
deployment-controller |
isvc-secondary-9e75a1-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-secondary-9e75a1-predictor-7b56c4bfc from 0 to 1 | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-9e75a1 |
UpdateFailed |
Failed to update status for InferenceService "isvc-secondary-9e75a1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-9e75a1": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-9e75a1 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-9e75a1": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5d63c390841fcd6358d7b4068789845000a8eff3dfe19f0f9b697dde4d31930a" already present on machine |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s |
Created |
Created container: storage-initializer |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-9e75a1 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-secondary-9e75a1-predictor-7b56c4bfc-kk54s_kserve-ci-e2e-test(3a8d798a-ef3b-456a-81a5-a5f4664590a3) | |
| (x6) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-9e75a1-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x6) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-9e75a1-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-9e75a1-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-9e75a1-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-9e75a1-predictor-7fd479db86-64jt7 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-9e75a1-predictor-7fd479db86-64jt7 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-9e75a1-predictor-7fd479db86-64jt7 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.24:8643/healthz": dial tcp 10.134.0.24:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-primary-9e75a1-predictor-7fd479db86-64jt7 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.24:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-279bfc |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-279bfc": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-279bfc |
UpdateFailed |
Failed to update status for InferenceService "isvc-init-fail-279bfc": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-279bfc": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-init-fail-279bfc-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-init-fail-279bfc-predictor-866d647b77 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-init-fail-279bfc-predictor-866d647b77 |
SuccessfulCreate |
Created pod: isvc-init-fail-279bfc-predictor-866d647b77-4nw6z | |
kserve-ci-e2e-test |
multus |
isvc-init-fail-279bfc-predictor-866d647b77-4nw6z |
AddedInterface |
Add eth0 [10.134.0.26/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-279bfc-predictor-866d647b77-4nw6z |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-279bfc-predictor-866d647b77-4nw6z |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-279bfc-predictor-866d647b77-4nw6z |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5d63c390841fcd6358d7b4068789845000a8eff3dfe19f0f9b697dde4d31930a" already present on machine |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-279bfc |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-279bfc-predictor-866d647b77-4nw6z |
Killing |
Stopping container storage-initializer | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-3f2fd-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-3f2fd-predictor-74d45cdc79 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-3f2fd-predictor-74d45cdc79 |
SuccessfulCreate |
Created pod: raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-3f2fd |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-3f2fd": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-3f2fd": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk |
AddedInterface |
Add eth0 [10.134.0.27/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5d63c390841fcd6358d7b4068789845000a8eff3dfe19f0f9b697dde4d31930a" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1745" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk |
Started |
Started container kube-rbac-proxy | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-3f2fd-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-3f2fd-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-3f2fd |
InferenceServiceReady |
InferenceService [raw-sklearn-3f2fd] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-3f2fd |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk |
Killing |
Stopping container kube-rbac-proxy | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-3f2fd-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-3f2fd-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.27:8643/healthz": dial tcp 10.134.0.27:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-3f2fd-predictor-74d45cdc79-nrngk |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.27:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5d63c390841fcd6358d7b4068789845000a8eff3dfe19f0f9b697dde4d31930a" already present on machine | |
kserve-ci-e2e-test |
multus |
raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp |
AddedInterface |
Add eth0 [10.134.0.28/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-24151 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-24151": the object has been modified; please apply your changes to the latest version and try again | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-24151 |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-runtime-24151": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-24151": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-runtime-24151-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-runtime-24151-predictor-674f4f6c9f from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-runtime-24151-predictor-674f4f6c9f |
SuccessfulCreate |
Created pod: raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1745" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp |
Started |
Started container kserve-container | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-24151-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-24151-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-24151 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-24151 |
InferenceServiceReady |
InferenceService [raw-sklearn-runtime-24151] is Ready | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp |
Killing |
Stopping container kube-rbac-proxy | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-24151-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-24151-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.28:8643/healthz": dial tcp 10.134.0.28:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-24151-predictor-674f4f6c9f-9gnmp |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.28:8080: connect: connection refused |