| Time | Namespace | Component | RelatedObject | Reason | Message |
|---|---|---|---|---|---|
kserve-ci-e2e-test |
isvc-canary-promote-v2-predictor-86766c4958-pnmcw |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pnmcw to ip-10-0-136-239.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-v2-predictor-6d8d4768d4-kx95s |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s to ip-10-0-137-64.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-d84a6-predictor-6478888887-8rwd8 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8 to ip-10-0-137-64.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-04101-68d8d6678c-qf9d8 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-04101-68d8d6678c-qf9d8 to ip-10-0-136-239.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk to ip-10-0-137-64.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-hpa-83c93-7f75bd67ff-7dhls |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-hpa-83c93-7f75bd67ff-7dhls to ip-10-0-136-239.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-predictor-86dc9fdb65-spv99 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99 to ip-10-0-137-64.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw to ip-10-0-137-64.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-be0d8-predictor-55f497b676-7vd95 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95 to ip-10-0-137-64.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs to ip-10-0-137-64.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-5bbb8c8cfc-4wdmc |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-4wdmc to ip-10-0-136-239.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-v2-predictor-77f778d864-rctkr |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rctkr to ip-10-0-136-239.ec2.internal | ||
kserve-ci-e2e-test |
isvc-primary-ede845-predictor-577854c8df-5cwfn |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn to ip-10-0-137-64.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh to ip-10-0-137-64.ec2.internal | ||
kserve-ci-e2e-test |
isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 to ip-10-0-137-64.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-predictor-d9759ffd5-g8nsd |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd to ip-10-0-137-64.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-v2-predictor-84f48b7656-wzk74 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74 to ip-10-0-137-64.ec2.internal | ||
kserve-ci-e2e-test |
isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv to ip-10-0-137-64.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29 to ip-10-0-137-64.ec2.internal | ||
kserve-ci-e2e-test |
isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj to ip-10-0-137-64.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v to ip-10-0-137-64.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-v2-predictor-86766c4958-djmzl |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl to ip-10-0-137-64.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-predictor-cb85cf55c-vx4z8 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8 to ip-10-0-137-64.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t to ip-10-0-137-64.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2 to ip-10-0-137-64.ec2.internal | ||
kserve-ci-e2e-test |
replicaset-controller |
isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464 |
SuccessfulCreate |
Created pod: isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-raw-sklearn-batcher-b19ff-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464 from 0 to 1 | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-b19ff |
UpdateFailed |
Failed to update status for InferenceService "isvc-raw-sklearn-batcher-b19ff": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-b19ff": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-b19ff |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-b19ff": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 |
AddedInterface |
Add eth0 [10.133.0.22/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-raw-sklearn-batcher-b19ff-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" in 12.538s (12.538s including waiting). Image size: 1555877302 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.359s (2.359s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-agent@sha256:efaf0bd4be5e59e0fd5df072416f565ae616956e4a23f814ea31d63b5efa6d4b" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:efaf0bd4be5e59e0fd5df072416f565ae616956e4a23f814ea31d63b5efa6d4b" in 2.09s (2.09s including waiting). Image size: 240168571 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 |
Started |
Started container agent | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-b19ff-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-b19ff-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-b19ff |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-b19ff |
InferenceServiceReady |
InferenceService [isvc-raw-sklearn-batcher-b19ff] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-04101-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-04101-predictor-54f484c8d4 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-04101-predictor-54f484c8d4 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-b19ff-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-b19ff-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-04101-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t |
AddedInterface |
Add eth0 [10.133.0.23/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-graph-raw-04101-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29 | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29 |
AddedInterface |
Add eth0 [10.133.0.24/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29 |
Started |
Started container storage-initializer | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29 |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
| (x3) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.22:8643/healthz": dial tcp 10.133.0.22:8643: connect: connection refused |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.22:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29 |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 18.143s (18.143s including waiting). Image size: 1307355896 bytes. | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-04101-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-04101-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-04101-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-04101-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-04101 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-04101 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-04101] is Ready | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-04101 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-04101 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-04101] is Ready | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-04101-68d8d6678c |
SuccessfulCreate |
Created pod: model-chainer-raw-04101-68d8d6678c-qf9d8 | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-04101 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-04101-68d8d6678c from 0 to 1 | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-04101 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-04101] is Ready | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-04101-68d8d6678c-qf9d8 |
AddedInterface |
Add eth0 [10.134.0.31/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-04101-68d8d6678c-qf9d8 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-router@sha256:24a06dcfbbe7f0d25115c5a79181136502c6b1ca34878eb57638d7eb57f020b2" | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-04101-68d8d6678c-qf9d8 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:24a06dcfbbe7f0d25115c5a79181136502c6b1ca34878eb57638d7eb57f020b2" in 2.424s (2.424s including waiting). Image size: 218226814 bytes. | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-04101-68d8d6678c-qf9d8 |
Created |
Created container: model-chainer-raw-04101 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-04101-68d8d6678c-qf9d8 |
Started |
Started container model-chainer-raw-04101 | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-04101-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-04101-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-04101-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-04101-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-04101-68d8d6678c-qf9d8 |
Killing |
Stopping container model-chainer-raw-04101 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2 |
Started |
Started container storage-initializer | |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.24:8080: connect: connection refused |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw |
AddedInterface |
Add eth0 [10.133.0.26/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-hpa-83c93-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.24:8643/healthz": dial tcp 10.133.0.24:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2 |
AddedInterface |
Add eth0 [10.133.0.25/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-hpa-83c93-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c from 0 to 1 | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.23:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.23:8643/healthz": dial tcp 10.133.0.23:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw |
Pulled |
Container image "kserve/xgbserver:latest" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw |
Created |
Created container: kube-rbac-proxy | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-04101-68d8d6678c-qf9d8 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-83c93-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-83c93-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-83c93-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-83c93-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-83c93 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-83c93 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-hpa-83c93] is Ready | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-83c93 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-83c93 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-hpa-83c93] is Ready | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-83c93-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-83c93-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-83c93 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-hpa-83c93] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-hpa-83c93 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-hpa-83c93-7f75bd67ff from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-hpa-83c93-7f75bd67ff |
SuccessfulCreate |
Created pod: model-chainer-raw-hpa-83c93-7f75bd67ff-7dhls | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-83c93-7f75bd67ff-7dhls |
Started |
Started container model-chainer-raw-hpa-83c93 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-83c93-7f75bd67ff-7dhls |
Created |
Created container: model-chainer-raw-hpa-83c93 | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-hpa-83c93-7f75bd67ff-7dhls |
AddedInterface |
Add eth0 [10.134.0.32/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-83c93-7f75bd67ff-7dhls |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:24a06dcfbbe7f0d25115c5a79181136502c6b1ca34878eb57638d7eb57f020b2" already present on machine | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-83c93-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-83c93-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-d84a6-predictor-6478888887 |
SuccessfulCreate |
Created pod: message-dumper-raw-d84a6-predictor-6478888887-8rwd8 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-d84a6 |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-d84a6": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-d84a6": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-d84a6-predictor-6478888887-8rwd8 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "message-dumper-raw-d84a6-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-d84a6-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-d84a6-predictor-6478888887 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-83c93-7f75bd67ff-7dhls |
Killing |
Stopping container model-chainer-raw-hpa-83c93 | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-d84a6-predictor-6478888887-8rwd8 |
AddedInterface |
Add eth0 [10.133.0.27/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.26:8643/healthz": dial tcp 10.133.0.26:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-d84a6-predictor-6478888887-8rwd8 |
Pulling |
Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.25:8643/healthz": dial tcp 10.133.0.25:8643: connect: connection refused | |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.26:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-d84a6-predictor-6478888887-8rwd8 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-d84a6-predictor-6478888887-8rwd8 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-d84a6-predictor-6478888887-8rwd8 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-d84a6-predictor-6478888887-8rwd8 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-d84a6-predictor-6478888887-8rwd8 |
Pulled |
Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.046s (1.046s including waiting). Image size: 14813193 bytes. | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-d84a6-predictor-6478888887-8rwd8 |
Created |
Created container: kube-rbac-proxy | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.25:8080: connect: connection refused |
| (x7) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-d84a6 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-d84a6 |
InferenceServiceReady |
InferenceService [message-dumper-raw-d84a6] is Ready | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-d84a6-predictor-5686cd4dcd |
SuccessfulCreate |
Created pod: isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-d84a6 |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-d84a6": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-d84a6": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-logger-raw-d84a6-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-d84a6-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-d84a6-predictor-5686cd4dcd from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v |
AddedInterface |
Add eth0 [10.133.0.28/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:efaf0bd4be5e59e0fd5df072416f565ae616956e4a23f814ea31d63b5efa6d4b" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v |
Created |
Created container: agent | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-83c93-7f75bd67ff-7dhls |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-d84a6-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-d84a6-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-d84a6-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-d84a6-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-d84a6-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-d84a6-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-d84a6 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-d84a6 |
InferenceServiceReady |
InferenceService [isvc-logger-raw-d84a6] is Ready | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-d84a6-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-d84a6-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-dca0e |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-scale-raw-dca0e": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scale-raw-dca0e": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-scale-raw-dca0e-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-scale-raw-dca0e-predictor-6d97785746 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-scale-raw-dca0e-predictor-6d97785746 |
SuccessfulCreate |
Created pod: isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-d84a6-predictor-6478888887-8rwd8 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-d84a6-predictor-6478888887-8rwd8 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh |
AddedInterface |
Add eth0 [10.133.0.29/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.28:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x4) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.28:8643/healthz": dial tcp 10.133.0.28:8643: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-dca0e-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-dca0e-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x8) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-dca0e-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x8) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-dca0e-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-dca0e |
InferenceServiceReady |
InferenceService [isvc-sklearn-scale-raw-dca0e] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-dca0e |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh |
Killing |
Stopping container kube-rbac-proxy | |
| (x17) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.29:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vx4z8 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-kx95s |
Created |
Created container: storage-initializer | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-create": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-create-v2-predictor-6d8d4768d4 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-v2-predictor-6d8d4768d4 |
SuccessfulCreate |
Created pod: isvc-canary-create-v2-predictor-6d8d4768d4-kx95s | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-kx95s |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vx4z8 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vx4z8 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-predictor-cb85cf55c |
SuccessfulCreate |
Created pod: isvc-canary-create-predictor-cb85cf55c-vx4z8 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-kx95s |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-predictor-cb85cf55c-vx4z8 |
AddedInterface |
Add eth0 [10.133.0.30/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-create-predictor-cb85cf55c from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-v2-predictor-6d8d4768d4-kx95s |
AddedInterface |
Add eth0 [10.133.0.31/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.29:8643/healthz": dial tcp 10.133.0.29:8643: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-kx95s |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-kx95s |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vx4z8 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-kx95s |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-kx95s |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-kx95s |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-kx95s |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vx4z8 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vx4z8 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vx4z8 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vx4z8 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vx4z8 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
| (x16) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
InferenceServiceReady |
InferenceService [isvc-canary-create] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-kx95s |
Killing |
Stopping container kube-rbac-proxy | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-kx95s |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.31:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vx4z8 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-kx95s |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vx4z8 |
Killing |
Stopping container kserve-container | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vx4z8 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.30:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vx4z8 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.30:8643/healthz": dial tcp 10.133.0.30:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-kx95s |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.31:8643/healthz": dial tcp 10.133.0.31:8643: connect: connection refused | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-promote": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-djmzl |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-v2-predictor-86766c4958-djmzl |
AddedInterface |
Add eth0 [10.133.0.33/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-predictor-d9759ffd5 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-predictor-d9759ffd5 |
SuccessfulCreate |
Created pod: isvc-canary-promote-predictor-d9759ffd5-g8nsd | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-djmzl |
Created |
Created container: storage-initializer | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-v2-predictor-86766c4958 |
SuccessfulCreate |
Created pod: isvc-canary-promote-v2-predictor-86766c4958-djmzl | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-v2-predictor-86766c4958 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-g8nsd |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-promote-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-g8nsd |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-g8nsd |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-djmzl |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-predictor-d9759ffd5-g8nsd |
AddedInterface |
Add eth0 [10.133.0.32/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-g8nsd |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-djmzl |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-djmzl |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-g8nsd |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-djmzl |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-djmzl |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-djmzl |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-djmzl |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-g8nsd |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-g8nsd |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-g8nsd |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-g8nsd |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-g8nsd |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-djmzl |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.33:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-g8nsd |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-v2-predictor-86766c4958 |
SuccessfulCreate |
Created pod: isvc-canary-promote-v2-predictor-86766c4958-pnmcw | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-g8nsd |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InferenceServiceNotReady |
InferenceService [isvc-canary-promote] is no longer Ready because of: PredictorReady, Ready, Stopped | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-v2-predictor-86766c4958-pnmcw |
AddedInterface |
Add eth0 [10.134.0.33/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-v2-predictor-86766c4958 from 1 to 2 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pnmcw |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" | |
| (x20) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-g8nsd |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.32:8643/healthz": dial tcp 10.133.0.32:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pnmcw |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" in 3.773s (3.773s including waiting). Image size: 300915674 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pnmcw |
Started |
Started container storage-initializer | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-g8nsd |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.32:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pnmcw |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pnmcw |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pnmcw |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pnmcw |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pnmcw |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pnmcw |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" in 12.742s (12.742s including waiting). Image size: 1555877302 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pnmcw |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pnmcw |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pnmcw |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.199s (2.199s including waiting). Image size: 211946088 bytes. | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pnmcw |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.33:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InferenceServiceReady |
InferenceService [isvc-canary-promote] is Ready |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pnmcw |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pnmcw |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-djmzl |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-djmzl |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-djmzl |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.33:8643/healthz": dial tcp 10.133.0.33:8643: connect: connection refused | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-predictor-5bbb8c8cfc from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-5bbb8c8cfc |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-v2-predictor-84f48b7656 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-v2-predictor-84f48b7656 |
SuccessfulCreate |
Created pod: isvc-canary-rollback-v2-predictor-84f48b7656-wzk74 | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk |
AddedInterface |
Add eth0 [10.133.0.34/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-wzk74 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-v2-predictor-84f48b7656-wzk74 |
AddedInterface |
Add eth0 [10.133.0.35/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-wzk74 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-wzk74 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-wzk74 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-wzk74 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-wzk74 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-wzk74 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-wzk74 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-wzk74 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-wzk74 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk |
Created |
Created container: kserve-container | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-wzk74 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.35:8080: connect: connection refused |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-predictor-5bbb8c8cfc from 1 to 2 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-wzk74 |
Killing |
Stopping container kserve-container | |
| (x3) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again |
| (x3) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-rollback": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again |
| (x19) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-wzk74 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-5bbb8c8cfc |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-5bbb8c8cfc-4wdmc | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-4wdmc |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-4wdmc |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-4wdmc |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-5bbb8c8cfc-4wdmc |
AddedInterface |
Add eth0 [10.134.0.34/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-wzk74 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.35:8643/healthz": dial tcp 10.133.0.35:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-4wdmc |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-4wdmc |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-4wdmc |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-4wdmc |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-4wdmc |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-4wdmc |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-4wdmc |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.34:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InferenceServiceReady |
InferenceService [isvc-canary-rollback] is Ready |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-4wdmc |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-4wdmc |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-4wdmc |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.34:8643/healthz": dial tcp 10.134.0.34:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.34:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.34:8643/healthz": dial tcp 10.133.0.34:8643: connect: connection refused | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-stop": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-stop-v2-predictor-77f778d864 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-v2-predictor-77f778d864 |
SuccessfulCreate |
Created pod: isvc-canary-stop-v2-predictor-77f778d864-rctkr | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-spv99 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-stop-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-predictor-86dc9fdb65 |
SuccessfulCreate |
Created pod: isvc-canary-stop-predictor-86dc9fdb65-spv99 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-stop-predictor-86dc9fdb65 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-rctkr |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-stop-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-rctkr |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-predictor-86dc9fdb65-spv99 |
AddedInterface |
Add eth0 [10.133.0.36/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-spv99 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-rctkr |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-rctkr |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-spv99 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-v2-predictor-77f778d864-rctkr |
AddedInterface |
Add eth0 [10.134.0.35/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-spv99 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-spv99 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-rctkr |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-rctkr |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-rctkr |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-rctkr |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-spv99 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-spv99 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-rctkr |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-spv99 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-spv99 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-spv99 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-rctkr |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InferenceServiceReady |
InferenceService [isvc-canary-stop] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-spv99 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-rctkr |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-rctkr |
Killing |
Stopping container kube-rbac-proxy | |
| (x20) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-spv99 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InferenceServiceNotReady |
InferenceService [isvc-canary-stop] is no longer Ready because of: PredictorReady, Ready | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-spv99 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.36:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-spv99 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.36:8643/healthz": dial tcp 10.133.0.36:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-rctkr |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.35:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-rctkr |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.35:8643/healthz": dial tcp 10.134.0.35:8643: connect: connection refused | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-ede845 |
UpdateFailed |
Failed to update status for InferenceService "isvc-primary-ede845": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-ede845": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-ede845-predictor-577854c8df-5cwfn |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-ede845 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-ede845": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-primary-ede845-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-primary-ede845-predictor-577854c8df from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-primary-ede845-predictor-577854c8df |
SuccessfulCreate |
Created pod: isvc-primary-ede845-predictor-577854c8df-5cwfn | |
kserve-ci-e2e-test |
multus |
isvc-primary-ede845-predictor-577854c8df-5cwfn |
AddedInterface |
Add eth0 [10.133.0.37/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-ede845-predictor-577854c8df-5cwfn |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-ede845-predictor-577854c8df-5cwfn |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-ede845-predictor-577854c8df-5cwfn |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-ede845-predictor-577854c8df-5cwfn |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-ede845-predictor-577854c8df-5cwfn |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-ede845-predictor-577854c8df-5cwfn |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-ede845-predictor-577854c8df-5cwfn |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-ede845-predictor-577854c8df-5cwfn |
Created |
Created container: kube-rbac-proxy | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-ede845-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-ede845-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-ede845 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-ede845 |
InferenceServiceReady |
InferenceService [isvc-primary-ede845] is Ready | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-ede845-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
replicaset-controller |
isvc-secondary-ede845-predictor-67bdf7dbbb |
SuccessfulCreate |
Created pod: isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-ede845-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
deployment-controller |
isvc-secondary-ede845-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-secondary-ede845-predictor-67bdf7dbbb from 0 to 1 | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-ede845 |
UpdateFailed |
Failed to update status for InferenceService "isvc-secondary-ede845": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-ede845": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-ede845 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-ede845": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
multus |
isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv |
AddedInterface |
Add eth0 [10.133.0.38/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv_kserve-ci-e2e-test(02a51eb9-3bfe-42e3-b353-afcd44834db0) | |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-ede845 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-ede845-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-ede845-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-ede845-predictor-577854c8df-5cwfn |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-ede845-predictor-577854c8df-5cwfn |
Killing |
Stopping container kserve-container | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-primary-ede845-predictor-577854c8df-5cwfn |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.37:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-primary-ede845-predictor-577854c8df-5cwfn |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.37:8643/healthz": dial tcp 10.133.0.37:8643: connect: connection refused | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-init-fail-30a290-predictor-7bbdbb8f5d |
SuccessfulCreate |
Created pod: isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj | |
kserve-ci-e2e-test |
deployment-controller |
isvc-init-fail-30a290-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-init-fail-30a290-predictor-7bbdbb8f5d from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-30a290 |
UpdateFailed |
Failed to update status for InferenceService "isvc-init-fail-30a290": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-30a290": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-30a290 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-30a290": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj |
AddedInterface |
Add eth0 [10.133.0.39/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-init-fail-30a290-predictor-serving-cert" not found | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-30a290 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj |
Killing |
Stopping container storage-initializer | |
kserve-ci-e2e-test |
multus |
raw-sklearn-be0d8-predictor-55f497b676-7vd95 |
AddedInterface |
Add eth0 [10.133.0.40/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-be0d8-predictor-55f497b676-7vd95 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-be0d8-predictor-55f497b676-7vd95 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-be0d8-predictor-55f497b676-7vd95 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-be0d8-predictor-55f497b676 |
SuccessfulCreate |
Created pod: raw-sklearn-be0d8-predictor-55f497b676-7vd95 | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-be0d8 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-be0d8": the object has been modified; please apply your changes to the latest version and try again | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-be0d8 |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-be0d8": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-be0d8": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-be0d8-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-be0d8-predictor-55f497b676 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-be0d8-predictor-55f497b676-7vd95 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-be0d8-predictor-55f497b676-7vd95 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-be0d8-predictor-55f497b676-7vd95 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-be0d8-predictor-55f497b676-7vd95 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-be0d8-predictor-55f497b676-7vd95 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-be0d8-predictor-55f497b676-7vd95 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-be0d8-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-be0d8-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-be0d8 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-be0d8 |
InferenceServiceReady |
InferenceService [raw-sklearn-be0d8] is Ready | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-be0d8-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-be0d8-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-be0d8-predictor-55f497b676-7vd95 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-be0d8-predictor-55f497b676-7vd95 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-be0d8-predictor-55f497b676-7vd95 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.40:8643/healthz": dial tcp 10.133.0.40:8643: connect: connection refused | |
| (x11) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-be0d8-predictor-55f497b676-7vd95 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.40:8080: connect: connection refused |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-runtime-8e468-predictor-845d8647f6 |
SuccessfulCreate |
Created pod: raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs | |
kserve-ci-e2e-test |
multus |
raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs |
AddedInterface |
Add eth0 [10.133.0.41/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:824dd4f70c10bfd12c956cfe8488915467e61b05d1a785bf64aa919cb1efc5da" already present on machine | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-8e468 |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-runtime-8e468": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-8e468": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-runtime-8e468-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-runtime-8e468-predictor-845d8647f6 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-8e468-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-8e468-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-8e468 |
InferenceServiceReady |
InferenceService [raw-sklearn-runtime-8e468] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-8e468 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-8e468-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-8e468-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.41:8643/healthz": dial tcp 10.133.0.41:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.41:8080: connect: connection refused |