| Time | Namespace | Component | RelatedObject | Reason | Message |
|---|---|---|---|---|---|
kserve-ci-e2e-test |
isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9 to ip-10-0-136-232.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8 to ip-10-0-136-232.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-8100c-854d568c9-r2lrk |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-8100c-854d568c9-r2lrk to ip-10-0-138-234.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-predictor-8bd7d86fd-gwc6t |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t to ip-10-0-136-232.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-103c4-predictor-86555d876f-r7wbl |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl to ip-10-0-136-232.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-v2-predictor-5b66dd5859-psnmx |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-psnmx to ip-10-0-138-234.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm to ip-10-0-136-232.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-predictor-7f74cfd478-4vq6v |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v to ip-10-0-136-232.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-6d775fdd5d-glsdr |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-glsdr to ip-10-0-138-234.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-hpa-6bdca-c66f8f685-9dhcp |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-hpa-6bdca-c66f8f685-9dhcp to ip-10-0-138-234.ec2.internal | ||
kserve-ci-e2e-test |
isvc-init-fail-972622-predictor-84968b6f47-xn72g |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-init-fail-972622-predictor-84968b6f47-xn72g to ip-10-0-138-234.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24 to ip-10-0-136-232.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5 to ip-10-0-136-232.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-runtime-700df-predictor-55c55666bc-mzbgh |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-runtime-700df-predictor-55c55666bc-mzbgh to ip-10-0-138-234.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-6d775fdd5d-bqvb4 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4 to ip-10-0-136-232.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz to ip-10-0-136-232.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-v2-predictor-5ff9f946fc-7h72k |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-7h72k to ip-10-0-138-234.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-v2-predictor-75c7994655-5v8h2 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2 to ip-10-0-136-232.ec2.internal | ||
kserve-ci-e2e-test |
isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r to ip-10-0-136-232.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr to ip-10-0-136-232.ec2.internal | ||
kserve-ci-e2e-test |
isvc-primary-cbae98-predictor-f485947c4-j2jzk |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-primary-cbae98-predictor-f485947c4-j2jzk to ip-10-0-138-234.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm to ip-10-0-136-232.ec2.internal | ||
kserve-ci-e2e-test |
isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t to ip-10-0-136-232.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd to ip-10-0-136-232.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-predictor-74874cc84b-hl285 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285 to ip-10-0-136-232.ec2.internal | ||
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t |
AddedInterface |
Add eth0 [10.134.0.25/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c |
SuccessfulCreate |
Created pod: isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
deployment-controller |
isvc-raw-sklearn-batcher-05405-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-05405 |
UpdateFailed |
Failed to update status for InferenceService "isvc-raw-sklearn-batcher-05405": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-05405": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-05405 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-05405": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" in 13.108s (13.108s including waiting). Image size: 1557885864 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t |
Pulling |
Pulling image "quay.io/opendatahub/kserve-agent@sha256:5eadda1fc9085300923462d30bccc33b4ca27b6d1bc72292ef033cbc042de960" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.521s (2.521s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:5eadda1fc9085300923462d30bccc33b4ca27b6d1bc72292ef033cbc042de960" in 2.77s (2.77s including waiting). Image size: 240177254 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t |
Started |
Started container agent | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-05405-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-05405-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-05405 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-05405 |
InferenceServiceReady |
InferenceService [isvc-raw-sklearn-batcher-05405] is Ready | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-05405-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-05405-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-8100c-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-8100c-predictor-76445879cf from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-8100c-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-8100c-predictor-76445879cf |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm |
AddedInterface |
Add eth0 [10.134.0.27/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd |
AddedInterface |
Add eth0 [10.134.0.26/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-graph-raw-8100c-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd |
Created |
Created container: kube-rbac-proxy | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.25:8080: connect: connection refused |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-8100c-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-8100c-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x3) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.25:8643/healthz": dial tcp 10.134.0.25:8643: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 18.092s (18.092s including waiting). Image size: 1307355899 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm |
Started |
Started container kube-rbac-proxy | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-8100c-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-8100c-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.26:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-8100c |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-8100c |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-8100c] is Ready | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-8100c |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-8100c |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-8100c] is Ready | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-8100c |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-8100c] is Ready | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-8100c-854d568c9-r2lrk |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-raw-8100c-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-8100c-854d568c9 |
SuccessfulCreate |
Created pod: model-chainer-raw-8100c-854d568c9-r2lrk | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-8100c |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-8100c-854d568c9 from 0 to 1 | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-8100c-854d568c9-r2lrk |
AddedInterface |
Add eth0 [10.132.0.23/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-8100c-854d568c9-r2lrk |
Pulling |
Pulling image "quay.io/opendatahub/kserve-router@sha256:d388f8d2997cdb455a73f33beea2e1aa8fe959b1d5682a0690a87a2319ba2789" | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-8100c-854d568c9-r2lrk |
Created |
Created container: model-chainer-raw-8100c | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-8100c-854d568c9-r2lrk |
Started |
Started container model-chainer-raw-8100c | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-8100c-854d568c9-r2lrk |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:d388f8d2997cdb455a73f33beea2e1aa8fe959b1d5682a0690a87a2319ba2789" in 2.339s (2.339s including waiting). Image size: 218233959 bytes. | |
| (x6) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-8100c-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x6) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-8100c-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-8100c-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-8100c-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-hpa-6bdca-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-6bdca |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-hpa-6bdca": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-hpa-6bdca": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-8100c-854d568c9-r2lrk |
Killing |
Stopping container model-chainer-raw-8100c | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24 |
AddedInterface |
Add eth0 [10.134.0.29/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-6bdca |
UpdateFailed |
Failed to update status for InferenceService "isvc-xgboost-graph-raw-hpa-6bdca": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-graph-raw-hpa-6bdca": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-hpa-6bdca-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz |
AddedInterface |
Add eth0 [10.134.0.28/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.27:8643/healthz": dial tcp 10.134.0.27:8643: connect: connection refused | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.27:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.26:8643/healthz": dial tcp 10.134.0.26:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24 |
Pulled |
Container image "kserve/xgbserver:latest" already present on machine | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-8100c-854d568c9-r2lrk |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-8100c-854d568c9-r2lrk |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.23:8080/readyz": dial tcp 10.132.0.23:8080: connect: connection refused | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-6bdca-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-6bdca-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-6bdca-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-6bdca-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-6bdca |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-6bdca |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-hpa-6bdca] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-6bdca |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-6bdca |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-hpa-6bdca] is Ready | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-6bdca-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-6bdca-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-6bdca-c66f8f685-9dhcp |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:d388f8d2997cdb455a73f33beea2e1aa8fe959b1d5682a0690a87a2319ba2789" already present on machine | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-hpa-6bdca-c66f8f685-9dhcp |
AddedInterface |
Add eth0 [10.132.0.24/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-hpa-6bdca-c66f8f685 |
SuccessfulCreate |
Created pod: model-chainer-raw-hpa-6bdca-c66f8f685-9dhcp | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-hpa-6bdca |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-hpa-6bdca-c66f8f685 from 0 to 1 | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-6bdca |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-hpa-6bdca] is Ready | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-6bdca-c66f8f685-9dhcp |
Created |
Created container: model-chainer-raw-hpa-6bdca | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-6bdca-c66f8f685-9dhcp |
Started |
Started container model-chainer-raw-hpa-6bdca | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-6bdca-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-6bdca-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-103c4-predictor-86555d876f-r7wbl |
AddedInterface |
Add eth0 [10.134.0.30/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-6bdca-c66f8f685-9dhcp |
Killing |
Stopping container model-chainer-raw-hpa-6bdca | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-103c4 |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-103c4": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-103c4": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-103c4-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-103c4-predictor-86555d876f from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-103c4-predictor-86555d876f |
SuccessfulCreate |
Created pod: message-dumper-raw-103c4-predictor-86555d876f-r7wbl | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-103c4-predictor-86555d876f-r7wbl |
Pulling |
Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-103c4-predictor-86555d876f-r7wbl |
Started |
Started container kserve-container | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.28:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-103c4-predictor-86555d876f-r7wbl |
Pulled |
Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.245s (1.245s including waiting). Image size: 14813193 bytes. | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-103c4-predictor-86555d876f-r7wbl |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-103c4-predictor-86555d876f-r7wbl |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-103c4-predictor-86555d876f-r7wbl |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-103c4-predictor-86555d876f-r7wbl |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.28:8643/healthz": dial tcp 10.134.0.28:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.29:8643/healthz": dial tcp 10.134.0.29:8643: connect: connection refused | |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.29:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-103c4 |
InferenceServiceReady |
InferenceService [message-dumper-raw-103c4] is Ready | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-103c4 |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-103c4": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-103c4": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr |
AddedInterface |
Add eth0 [10.134.0.31/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-103c4-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-103c4-predictor-95c8bff79 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine | |
| (x8) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-103c4 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-103c4-predictor-95c8bff79 |
SuccessfulCreate |
Created pod: isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:5eadda1fc9085300923462d30bccc33b4ca27b6d1bc72292ef033cbc042de960" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr |
Created |
Created container: agent | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-6bdca-c66f8f685-9dhcp |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-103c4-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-103c4-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-103c4-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-103c4-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-103c4 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-103c4 |
InferenceServiceReady |
InferenceService [isvc-logger-raw-103c4] is Ready | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-103c4-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-103c4-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-103c4-predictor-86555d876f-r7wbl |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-6f3c9 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scale-raw-6f3c9": the object has been modified; please apply your changes to the latest version and try again | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-6f3c9 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-scale-raw-6f3c9": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scale-raw-6f3c9": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-scale-raw-6f3c9-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf |
SuccessfulCreate |
Created pod: isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8 | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-103c4-predictor-86555d876f-r7wbl |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8 |
AddedInterface |
Add eth0 [10.134.0.32/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8 |
Started |
Started container kube-rbac-proxy | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.31:8080: connect: connection refused |
| (x3) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.31:8643/healthz": dial tcp 10.134.0.31:8643: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-6f3c9-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-6f3c9-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x8) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-6f3c9-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x8) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-6f3c9-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-6f3c9 |
InferenceServiceReady |
InferenceService [isvc-sklearn-scale-raw-6f3c9] is Ready | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-6f3c9 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8 |
Killing |
Stopping container kube-rbac-proxy | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-75c7994655-5v8h2 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-predictor-7f74cfd478 |
SuccessfulCreate |
Created pod: isvc-canary-create-predictor-7f74cfd478-4vq6v | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-create-v2-predictor-75c7994655 from 0 to 1 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-create-predictor-7f74cfd478 from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-v2-predictor-75c7994655-5v8h2 |
AddedInterface |
Add eth0 [10.134.0.34/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-75c7994655-5v8h2 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-create": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-75c7994655-5v8h2 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-v2-predictor-75c7994655 |
SuccessfulCreate |
Created pod: isvc-canary-create-v2-predictor-75c7994655-5v8h2 | |
| (x18) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.32:8080: connect: connection refused |
kserve-ci-e2e-test |
multus |
isvc-canary-create-predictor-7f74cfd478-4vq6v |
AddedInterface |
Add eth0 [10.134.0.33/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7f74cfd478-4vq6v |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7f74cfd478-4vq6v |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7f74cfd478-4vq6v |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.32:8643/healthz": dial tcp 10.134.0.32:8643: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-75c7994655-5v8h2 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7f74cfd478-4vq6v |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7f74cfd478-4vq6v |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7f74cfd478-4vq6v |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-75c7994655-5v8h2 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-75c7994655-5v8h2 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-75c7994655-5v8h2 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-75c7994655-5v8h2 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7f74cfd478-4vq6v |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-75c7994655-5v8h2 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7f74cfd478-4vq6v |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7f74cfd478-4vq6v |
Created |
Created container: kube-rbac-proxy | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7f74cfd478-4vq6v |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.33:8080: connect: connection refused |
| (x15) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
InferenceServiceReady |
InferenceService [isvc-canary-create] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-75c7994655-5v8h2 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7f74cfd478-4vq6v |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-75c7994655-5v8h2 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7f74cfd478-4vq6v |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7f74cfd478-4vq6v |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.33:8643/healthz": dial tcp 10.134.0.33:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-75c7994655-5v8h2 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.34:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-75c7994655-5v8h2 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.34:8643/healthz": dial tcp 10.134.0.34:8643: connect: connection refused | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9 |
AddedInterface |
Add eth0 [10.134.0.36/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-v2-predictor-5ff9f946fc from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-v2-predictor-5ff9f946fc |
SuccessfulCreate |
Created pod: isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-predictor-74874cc84b from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-predictor-74874cc84b |
SuccessfulCreate |
Created pod: isvc-canary-promote-predictor-74874cc84b-hl285 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-74874cc84b-hl285 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-promote-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-predictor-74874cc84b-hl285 |
AddedInterface |
Add eth0 [10.134.0.35/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-74874cc84b-hl285 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-74874cc84b-hl285 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-74874cc84b-hl285 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-74874cc84b-hl285 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-74874cc84b-hl285 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-74874cc84b-hl285 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-74874cc84b-hl285 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-74874cc84b-hl285 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-74874cc84b-hl285 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InferenceServiceNotReady |
InferenceService [isvc-canary-promote] is no longer Ready because of: PredictorReady, Ready, Stopped | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-v2-predictor-5ff9f946fc |
SuccessfulCreate |
Created pod: isvc-canary-promote-v2-predictor-5ff9f946fc-7h72k | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-74874cc84b-hl285 |
Killing |
Stopping container kserve-container | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-promote": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-74874cc84b-hl285 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-v2-predictor-5ff9f946fc from 1 to 2 | |
| (x20) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-v2-predictor-5ff9f946fc-7h72k |
AddedInterface |
Add eth0 [10.132.0.25/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-7h72k |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-74874cc84b-hl285 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.35:8643/healthz": dial tcp 10.134.0.35:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-74874cc84b-hl285 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.35:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-7h72k |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" in 3.468s (3.468s including waiting). Image size: 302292931 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-7h72k |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-7h72k |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-7h72k |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-7h72k |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" in 13.874s (13.874s including waiting). Image size: 1557885864 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-7h72k |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-7h72k |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-7h72k |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-7h72k |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-7h72k |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-7h72k |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.254s (2.254s including waiting). Image size: 211946088 bytes. | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-7h72k |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.25:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InferenceServiceReady |
InferenceService [isvc-canary-promote] is Ready |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-7h72k |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-7h72k |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9 |
Killing |
Stopping container kserve-container | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.36:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.36:8643/healthz": dial tcp 10.134.0.36:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-5ff9f946fc-7h72k |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.25:8643/healthz": dial tcp 10.132.0.25:8643: connect: connection refused | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-6d775fdd5d |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-6d775fdd5d-glsdr | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-predictor-6d775fdd5d from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-v2-predictor-5b9f7db4c5 |
SuccessfulCreate |
Created pod: isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-v2-predictor-5b9f7db4c5 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-glsdr |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-glsdr |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-glsdr |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-6d775fdd5d-glsdr |
AddedInterface |
Add eth0 [10.132.0.26/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm |
AddedInterface |
Add eth0 [10.134.0.37/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-glsdr |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-glsdr |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-glsdr |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-glsdr |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-glsdr |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-glsdr |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-6d775fdd5d |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-6d775fdd5d-bqvb4 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-bqvb4 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-bqvb4 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-bqvb4 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InferenceServiceNotReady |
InferenceService [isvc-canary-rollback] is no longer Ready because of: PredictorReady, Ready, Stopped | |
| (x3) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-rollback": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-6d775fdd5d-bqvb4 |
AddedInterface |
Add eth0 [10.134.0.38/23] from ovn-kubernetes | |
| (x19) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-predictor-6d775fdd5d from 1 to 2 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.37:8643/healthz": dial tcp 10.134.0.37:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.37:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-bqvb4 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-bqvb4 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-bqvb4 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-bqvb4 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-bqvb4 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-bqvb4 |
Started |
Started container kserve-container | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-bqvb4 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.38:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InferenceServiceReady |
InferenceService [isvc-canary-rollback] is Ready |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-glsdr |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-glsdr |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-bqvb4 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-bqvb4 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-glsdr |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.26:8643/healthz": dial tcp 10.132.0.26:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-glsdr |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.26:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-6d775fdd5d-bqvb4 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.38:8643/healthz": dial tcp 10.134.0.38:8643: connect: connection refused | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-stop": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-predictor-8bd7d86fd |
SuccessfulCreate |
Created pod: isvc-canary-stop-predictor-8bd7d86fd-gwc6t | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-stop-predictor-8bd7d86fd from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5b66dd5859-psnmx |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-stop-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-stop-v2-predictor-5b66dd5859 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-8bd7d86fd-gwc6t |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-stop-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-v2-predictor-5b66dd5859 |
SuccessfulCreate |
Created pod: isvc-canary-stop-v2-predictor-5b66dd5859-psnmx | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-8bd7d86fd-gwc6t |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5b66dd5859-psnmx |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5b66dd5859-psnmx |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5b66dd5859-psnmx |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-v2-predictor-5b66dd5859-psnmx |
AddedInterface |
Add eth0 [10.132.0.27/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-8bd7d86fd-gwc6t |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-8bd7d86fd-gwc6t |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-predictor-8bd7d86fd-gwc6t |
AddedInterface |
Add eth0 [10.134.0.39/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-8bd7d86fd-gwc6t |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5b66dd5859-psnmx |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5b66dd5859-psnmx |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5b66dd5859-psnmx |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5b66dd5859-psnmx |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5b66dd5859-psnmx |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5b66dd5859-psnmx |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-8bd7d86fd-gwc6t |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-8bd7d86fd-gwc6t |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-8bd7d86fd-gwc6t |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-8bd7d86fd-gwc6t |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-8bd7d86fd-gwc6t |
Started |
Started container kserve-container | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5b66dd5859-psnmx |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.27:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InferenceServiceReady |
InferenceService [isvc-canary-stop] is Ready | |
| (x21) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InferenceServiceNotReady |
InferenceService [isvc-canary-stop] is no longer Ready because of: PredictorReady, Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5b66dd5859-psnmx |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5b66dd5859-psnmx |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-8bd7d86fd-gwc6t |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-8bd7d86fd-gwc6t |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-8bd7d86fd-gwc6t |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.39:8643/healthz": dial tcp 10.134.0.39:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-5b66dd5859-psnmx |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.27:8643/healthz": dial tcp 10.132.0.27:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-8bd7d86fd-gwc6t |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.39:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-cbae98 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-cbae98": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-cbae98 |
UpdateFailed |
Failed to update status for InferenceService "isvc-primary-cbae98": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-cbae98": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
deployment-controller |
isvc-primary-cbae98-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-primary-cbae98-predictor-f485947c4 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-primary-cbae98-predictor-f485947c4 |
SuccessfulCreate |
Created pod: isvc-primary-cbae98-predictor-f485947c4-j2jzk | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-cbae98-predictor-f485947c4-j2jzk |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-cbae98-predictor-f485947c4-j2jzk |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-primary-cbae98-predictor-f485947c4-j2jzk |
AddedInterface |
Add eth0 [10.132.0.28/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-cbae98-predictor-f485947c4-j2jzk |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-cbae98-predictor-f485947c4-j2jzk |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-primary-cbae98-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-cbae98-predictor-f485947c4-j2jzk |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-cbae98-predictor-f485947c4-j2jzk |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-cbae98-predictor-f485947c4-j2jzk |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-cbae98-predictor-f485947c4-j2jzk |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-cbae98-predictor-f485947c4-j2jzk |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-cbae98-predictor-f485947c4-j2jzk |
Created |
Created container: kube-rbac-proxy | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-cbae98-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-cbae98-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-primary-cbae98-predictor-f485947c4-j2jzk |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.28:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-cbae98 |
InferenceServiceReady |
InferenceService [isvc-primary-cbae98] is Ready | |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-cbae98 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
replicaset-controller |
isvc-secondary-cbae98-predictor-6dcc6c88bf |
SuccessfulCreate |
Created pod: isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r | |
kserve-ci-e2e-test |
deployment-controller |
isvc-secondary-cbae98-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-secondary-cbae98-predictor-6dcc6c88bf from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-cbae98 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-cbae98": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-cbae98 |
UpdateFailed |
Failed to update status for InferenceService "isvc-secondary-cbae98": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-cbae98": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r |
AddedInterface |
Add eth0 [10.134.0.40/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-cbae98-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-cbae98-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r_kserve-ci-e2e-test(dcf3d8ad-1248-4e38-8984-c1499c3e71f8) | |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-cbae98 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-cbae98-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-cbae98-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-cbae98-predictor-f485947c4-j2jzk |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-cbae98-predictor-f485947c4-j2jzk |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-cbae98-predictor-f485947c4-j2jzk |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.28:8643/healthz": dial tcp 10.132.0.28:8643: connect: connection refused | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-init-fail-972622-predictor-84968b6f47 |
SuccessfulCreate |
Created pod: isvc-init-fail-972622-predictor-84968b6f47-xn72g | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-972622 |
UpdateFailed |
Failed to update status for InferenceService "isvc-init-fail-972622": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-972622": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-init-fail-972622-predictor-84968b6f47-xn72g |
AddedInterface |
Add eth0 [10.132.0.29/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
isvc-init-fail-972622-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-init-fail-972622-predictor-84968b6f47 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-972622 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-972622": the object has been modified; please apply your changes to the latest version and try again | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-972622-predictor-84968b6f47-xn72g |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-972622-predictor-84968b6f47-xn72g |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-972622-predictor-84968b6f47-xn72g |
Started |
Started container storage-initializer |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-972622 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-972622-predictor-84968b6f47-xn72g |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-init-fail-972622-predictor-84968b6f47-xn72g_kserve-ci-e2e-test(fb1f7063-8522-4e6d-8fd3-f164fd3b4441) | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "raw-sklearn-1170d-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-1170d-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-1170d-predictor-7b76dc8fff from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-1170d-predictor-7b76dc8fff |
SuccessfulCreate |
Created pod: raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5 | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-1170d |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-1170d": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-1170d": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5 |
AddedInterface |
Add eth0 [10.134.0.41/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5 |
Started |
Started container kserve-container | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-1170d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-1170d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-1170d |
InferenceServiceReady |
InferenceService [raw-sklearn-1170d] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-1170d |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-1170d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-1170d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.41:8643/healthz": dial tcp 10.134.0.41:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.41:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-700df |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-700df": the object has been modified; please apply your changes to the latest version and try again | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-700df |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-runtime-700df": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-700df": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-runtime-700df-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-runtime-700df-predictor-55c55666bc from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-runtime-700df-predictor-55c55666bc |
SuccessfulCreate |
Created pod: raw-sklearn-runtime-700df-predictor-55c55666bc-mzbgh | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-700df-predictor-55c55666bc-mzbgh |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:4bc99d27049df235a3e7474685f2095e74c6eed89dc3da2f17a1c3e85c3fa894" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-700df-predictor-55c55666bc-mzbgh |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-700df-predictor-55c55666bc-mzbgh |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
raw-sklearn-runtime-700df-predictor-55c55666bc-mzbgh |
AddedInterface |
Add eth0 [10.132.0.30/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-700df-predictor-55c55666bc-mzbgh |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-700df-predictor-55c55666bc-mzbgh |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1716" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-700df-predictor-55c55666bc-mzbgh |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-700df-predictor-55c55666bc-mzbgh |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-700df-predictor-55c55666bc-mzbgh |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-700df-predictor-55c55666bc-mzbgh |
Started |
Started container kserve-container | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-700df-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-700df-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-700df |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-700df |
InferenceServiceReady |
InferenceService [raw-sklearn-runtime-700df] is Ready | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-700df-predictor-55c55666bc-mzbgh |
Killing |
Stopping container kube-rbac-proxy | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-700df-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-700df-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-700df-predictor-55c55666bc-mzbgh |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-700df-predictor-55c55666bc-mzbgh |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.30:8643/healthz": dial tcp 10.132.0.30:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-700df-predictor-55c55666bc-mzbgh |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.30:8080: connect: connection refused |