| Time | Namespace | Component | RelatedObject | Reason | Message |
|---|---|---|---|---|---|
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s to ip-10-0-131-253.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6 to ip-10-0-131-253.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj to ip-10-0-131-253.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-hpa-1559b-855c7c546-q7gkm |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-hpa-1559b-855c7c546-q7gkm to ip-10-0-134-38.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-6148e-5c4fc4b9b6-4d7b9 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-6148e-5c4fc4b9b6-4d7b9 to ip-10-0-134-38.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn to ip-10-0-131-253.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x to ip-10-0-131-253.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-5bbb8c8cfc-dr7c2 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-dr7c2 to ip-10-0-139-153.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2 to ip-10-0-131-253.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-predictor-86dc9fdb65-jb7wd |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-jb7wd to ip-10-0-131-253.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-predictor-cb85cf55c-vzbc6 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vzbc6 to ip-10-0-131-253.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-v2-predictor-77f778d864-5hxnt |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-5hxnt to ip-10-0-131-253.ec2.internal | ||
kserve-ci-e2e-test |
isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k to ip-10-0-131-253.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-4bc18-predictor-595655c87f-p2psf |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-4bc18-predictor-595655c87f-p2psf to ip-10-0-131-253.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b to ip-10-0-131-253.ec2.internal | ||
kserve-ci-e2e-test |
isvc-primary-56d0f5-predictor-fb478888f-prb6c |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-primary-56d0f5-predictor-fb478888f-prb6c to ip-10-0-131-253.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs to ip-10-0-131-253.ec2.internal | ||
kserve-ci-e2e-test |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp to ip-10-0-131-253.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv to ip-10-0-131-253.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc to ip-10-0-131-253.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-v2-predictor-86766c4958-sz22f |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-sz22f to ip-10-0-131-253.ec2.internal | ||
kserve-ci-e2e-test |
isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt to ip-10-0-131-253.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-predictor-d9759ffd5-ppssc |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppssc to ip-10-0-131-253.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn to ip-10-0-131-253.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-v2-predictor-86766c4958-pkjv5 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pkjv5 to ip-10-0-139-153.ec2.internal | ||
kserve-ci-e2e-test |
replicaset-controller |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8 |
SuccessfulCreate |
Created pod: isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp | |
kserve-ci-e2e-test |
deployment-controller |
isvc-raw-sklearn-batcher-05e8c-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8 from 0 to 1 | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-05e8c |
UpdateFailed |
Failed to update status for InferenceService "isvc-raw-sklearn-batcher-05e8c": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-05e8c": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-05e8c |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-05e8c": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-raw-sklearn-batcher-05e8c-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" | |
kserve-ci-e2e-test |
multus |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
AddedInterface |
Add eth0 [10.133.0.30/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" in 3.063s (3.063s including waiting). Image size: 300981723 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" in 13.908s (13.908s including waiting). Image size: 1555729331 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
Pulling |
Pulling image "quay.io/opendatahub/kserve-agent@sha256:bf42462edec7da298c1b007872394204ebe10a2c0c0d143725247988fb0a7324" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.767s (2.767s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:bf42462edec7da298c1b007872394204ebe10a2c0c0d143725247988fb0a7324" in 2.624s (2.624s including waiting). Image size: 240177278 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
Created |
Created container: agent | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-05e8c-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-05e8c-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-05e8c |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-05e8c |
InferenceServiceReady |
InferenceService [isvc-raw-sklearn-batcher-05e8c] is Ready | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-05e8c-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-05e8c-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-6148e-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.30:8643/healthz": EOF | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-6148e |
UpdateFailed |
Failed to update status for InferenceService "isvc-xgboost-graph-raw-6148e": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-graph-raw-6148e": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-graph-raw-6148e-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-6148e |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-6148e": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-6148e": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-6148e-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988 from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s |
AddedInterface |
Add eth0 [10.133.0.32/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2 |
AddedInterface |
Add eth0 [10.133.0.31/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.30:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2 |
Created |
Created container: kserve-container | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-05e8c-predictor-6587dc5bd8-992jp |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.30:8643/healthz": dial tcp 10.133.0.30:8643: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 19.568s (19.568s including waiting). Image size: 1307355896 bytes. | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-6148e-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-6148e-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-6148e-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-6148e-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.31:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-6148e |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-6148e] is Ready | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-6148e |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-6148e |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-6148e |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-6148e] is Ready | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-6148e-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-6148e-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-6148e |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-6148e] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-6148e |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-6148e-5c4fc4b9b6 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-6148e-5c4fc4b9b6 |
SuccessfulCreate |
Created pod: model-chainer-raw-6148e-5c4fc4b9b6-4d7b9 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-6148e-5c4fc4b9b6-4d7b9 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-router@sha256:3f8b4d8bbc07de8aec3f29b089d69c216cddd2ca68e10f109f830f4b0fdd9ee1" | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-6148e-5c4fc4b9b6-4d7b9 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-raw-6148e-serving-cert" not found | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-6148e-5c4fc4b9b6-4d7b9 |
AddedInterface |
Add eth0 [10.132.0.27/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-6148e-5c4fc4b9b6-4d7b9 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:3f8b4d8bbc07de8aec3f29b089d69c216cddd2ca68e10f109f830f4b0fdd9ee1" in 2.082s (2.082s including waiting). Image size: 218232445 bytes. | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-6148e-5c4fc4b9b6-4d7b9 |
Created |
Created container: model-chainer-raw-6148e | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-6148e-5c4fc4b9b6-4d7b9 |
Started |
Started container model-chainer-raw-6148e | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-6148e-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-6148e-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-6148e-5c4fc4b9b6-4d7b9 |
Killing |
Stopping container model-chainer-raw-6148e | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-graph-raw-hpa-1559b-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-1559b |
UpdateFailed |
Failed to update status for InferenceService "isvc-xgboost-graph-raw-hpa-1559b": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-graph-raw-hpa-1559b": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-hpa-1559b-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9 from 0 to 1 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-hpa-1559b-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-graph-raw-hpa-1559b-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.32:8643/healthz": dial tcp 10.133.0.32:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s |
Killing |
Stopping container kserve-container | |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-6148e-predictor-85bd99f6bf-dsf2s |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.32:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-1559b |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-hpa-1559b": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-hpa-1559b": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x |
AddedInterface |
Add eth0 [10.133.0.34/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc |
AddedInterface |
Add eth0 [10.133.0.33/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-6148e-predictor-84cbc7c988-cd2d2 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.31:8643/healthz": dial tcp 10.133.0.31:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x |
Pulled |
Container image "kserve/xgbserver:latest" already present on machine | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-1559b-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-1559b-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-6148e-5c4fc4b9b6-4d7b9 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-1559b-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-1559b-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-1559b |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-hpa-1559b] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-1559b |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-1559b |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-1559b |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-hpa-1559b] is Ready | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-1559b-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-1559b-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-hpa-1559b |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-hpa-1559b-855c7c546 from 0 to 1 | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-1559b |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-hpa-1559b] is Ready | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-1559b-855c7c546-q7gkm |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-raw-hpa-1559b-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-hpa-1559b-855c7c546 |
SuccessfulCreate |
Created pod: model-chainer-raw-hpa-1559b-855c7c546-q7gkm | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-hpa-1559b-855c7c546-q7gkm |
AddedInterface |
Add eth0 [10.132.0.28/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-1559b-855c7c546-q7gkm |
Started |
Started container model-chainer-raw-hpa-1559b | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-1559b-855c7c546-q7gkm |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:3f8b4d8bbc07de8aec3f29b089d69c216cddd2ca68e10f109f830f4b0fdd9ee1" already present on machine | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-1559b-855c7c546-q7gkm |
Created |
Created container: model-chainer-raw-hpa-1559b | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-1559b-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-1559b-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-4bc18 |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-4bc18": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-4bc18": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "message-dumper-raw-4bc18-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-4bc18-predictor-5688fcbbf8 |
SuccessfulCreate |
Created pod: message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-1559b-855c7c546-q7gkm |
Killing |
Stopping container model-chainer-raw-hpa-1559b | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-4bc18-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-4bc18-predictor-5688fcbbf8 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn |
Pulling |
Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn |
AddedInterface |
Add eth0 [10.133.0.35/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.33:8643/healthz": dial tcp 10.133.0.33:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn |
Pulled |
Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.149s (1.149s including waiting). Image size: 14813193 bytes. | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-1559b-predictor-7b98f6788-zgxzc |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.33:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn |
Started |
Started container kube-rbac-proxy | |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.34:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-1559b-predictor-59dcccf4d9-xj47x |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.34:8643/healthz": dial tcp 10.133.0.34:8643: connect: connection refused | |
| (x6) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-4bc18 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-4bc18 |
InferenceServiceReady |
InferenceService [message-dumper-raw-4bc18] is Ready | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-4bc18-predictor-595655c87f-p2psf |
AddedInterface |
Add eth0 [10.133.0.36/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-4bc18-predictor-595655c87f-p2psf |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-4bc18-predictor-595655c87f-p2psf |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-4bc18-predictor-595655c87f-p2psf |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" already present on machine | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-4bc18 |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-4bc18": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-4bc18": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-4bc18-predictor-595655c87f |
SuccessfulCreate |
Created pod: isvc-logger-raw-4bc18-predictor-595655c87f-p2psf | |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-4bc18-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-4bc18-predictor-595655c87f from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-4bc18-predictor-595655c87f-p2psf |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-4bc18-predictor-595655c87f-p2psf |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-4bc18-predictor-595655c87f-p2psf |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-4bc18-predictor-595655c87f-p2psf |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-4bc18-predictor-595655c87f-p2psf |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-4bc18-predictor-595655c87f-p2psf |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:bf42462edec7da298c1b007872394204ebe10a2c0c0d143725247988fb0a7324" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-4bc18-predictor-595655c87f-p2psf |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-4bc18-predictor-595655c87f-p2psf |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-4bc18-predictor-595655c87f-p2psf |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-1559b-855c7c546-q7gkm |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-4bc18-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-4bc18-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-4bc18-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-4bc18-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-4bc18-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-4bc18-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-4bc18 |
InferenceServiceReady |
InferenceService [isvc-logger-raw-4bc18] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-4bc18 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-4bc18-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-4bc18-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-4bc18-predictor-595655c87f-p2psf |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-scale-raw-dcfcf-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98 |
SuccessfulCreate |
Created pod: isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-scale-raw-dcfcf-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-4bc18-predictor-595655c87f-p2psf |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-4bc18-predictor-595655c87f-p2psf |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-dcfcf |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-scale-raw-dcfcf": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scale-raw-dcfcf": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-4bc18-predictor-5688fcbbf8-qxqpn |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv |
AddedInterface |
Add eth0 [10.133.0.37/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv |
Created |
Created container: kube-rbac-proxy | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-4bc18-predictor-595655c87f-p2psf |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-4bc18-predictor-595655c87f-p2psf |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.36:8080: connect: connection refused |
| (x4) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-4bc18-predictor-595655c87f-p2psf |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.36:8643/healthz": dial tcp 10.133.0.36:8643: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-dcfcf-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-dcfcf-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x8) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-dcfcf-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x8) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-dcfcf-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-dcfcf |
InferenceServiceReady |
InferenceService [isvc-sklearn-scale-raw-dcfcf] is Ready | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-dcfcf |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv |
Killing |
Stopping container kube-rbac-proxy | |
| (x17) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.37:8080: connect: connection refused |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-create-predictor-cb85cf55c from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vzbc6 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vzbc6 |
Started |
Started container storage-initializer | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-create": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-predictor-cb85cf55c |
SuccessfulCreate |
Created pod: isvc-canary-create-predictor-cb85cf55c-vzbc6 | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6 |
AddedInterface |
Add eth0 [10.133.0.39/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-predictor-cb85cf55c-vzbc6 |
AddedInterface |
Add eth0 [10.133.0.38/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-v2-predictor-6d8d4768d4 |
SuccessfulCreate |
Created pod: isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-create-v2-predictor-6d8d4768d4 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vzbc6 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" already present on machine | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-dcfcf-predictor-56b8756c98-67zhv |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.37:8643/healthz": dial tcp 10.133.0.37:8643: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vzbc6 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vzbc6 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vzbc6 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vzbc6 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vzbc6 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vzbc6 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
InferenceServiceReady |
InferenceService [isvc-canary-create] is Ready | |
| (x17) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vzbc6 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vzbc6 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vzbc6 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.38:8643/healthz": dial tcp 10.133.0.38:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-vzbc6 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.38:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.39:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-rbdx6 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.39:8643/healthz": dial tcp 10.133.0.39:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-sz22f |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-sz22f |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" already present on machine | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-v2-predictor-86766c4958 |
SuccessfulCreate |
Created pod: isvc-canary-promote-v2-predictor-86766c4958-sz22f | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-predictor-d9759ffd5 |
SuccessfulCreate |
Created pod: isvc-canary-promote-predictor-d9759ffd5-ppssc | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-predictor-d9759ffd5 from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-v2-predictor-86766c4958-sz22f |
AddedInterface |
Add eth0 [10.133.0.41/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-v2-predictor-86766c4958 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-sz22f |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-ppssc |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-promote-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-ppssc |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-predictor-d9759ffd5-ppssc |
AddedInterface |
Add eth0 [10.133.0.40/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-ppssc |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-ppssc |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-ppssc |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-sz22f |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-sz22f |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-sz22f |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-ppssc |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-ppssc |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-ppssc |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-ppssc |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-ppssc |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-sz22f |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-sz22f |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-sz22f |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-v2-predictor-86766c4958-pkjv5 |
AddedInterface |
Add eth0 [10.134.0.17/23] from ovn-kubernetes | |
| (x3) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-v2-predictor-86766c4958 from 1 to 2 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-v2-predictor-86766c4958 |
SuccessfulCreate |
Created pod: isvc-canary-promote-v2-predictor-86766c4958-pkjv5 | |
| (x3) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-promote": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again |
| (x19) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pkjv5 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pkjv5 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-ppssc |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pkjv5 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-ppssc |
Killing |
Stopping container kserve-container | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-ppssc |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.40:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-ppssc |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.40:8643/healthz": dial tcp 10.133.0.40:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pkjv5 |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pkjv5 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pkjv5 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pkjv5 |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pkjv5 |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" in 13.879s (13.879s including waiting). Image size: 1555729331 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pkjv5 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pkjv5 |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.381s (2.381s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pkjv5 |
Created |
Created container: kube-rbac-proxy | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pkjv5 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.17:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InferenceServiceReady |
InferenceService [isvc-canary-promote] is Ready |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-sz22f |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-sz22f |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pkjv5 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pkjv5 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-pkjv5 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.17:8643/healthz": dial tcp 10.134.0.17:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-sz22f |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.41:8643/healthz": dial tcp 10.133.0.41:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-sz22f |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.41:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-v2-predictor-84f48b7656 from 0 to 1 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-predictor-5bbb8c8cfc from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-v2-predictor-84f48b7656 |
SuccessfulCreate |
Created pod: isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-5bbb8c8cfc |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs |
AddedInterface |
Add eth0 [10.133.0.43/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn |
AddedInterface |
Add eth0 [10.133.0.42/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs |
Killing |
Stopping container kube-rbac-proxy | |
| (x3) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-predictor-5bbb8c8cfc from 1 to 2 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-5bbb8c8cfc |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-5bbb8c8cfc-dr7c2 | |
| (x19) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x3) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-rollback": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-5bbb8c8cfc-dr7c2 |
AddedInterface |
Add eth0 [10.134.0.18/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-dr7c2 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-dr7c2 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-dr7c2 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" already present on machine | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.43:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-8k6zs |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.43:8643/healthz": dial tcp 10.133.0.43:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-dr7c2 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-dr7c2 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-dr7c2 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-dr7c2 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-dr7c2 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-dr7c2 |
Created |
Created container: kserve-container | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-dr7c2 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.18:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InferenceServiceReady |
InferenceService [isvc-canary-rollback] is Ready |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-dr7c2 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-dr7c2 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-dr7c2 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.18:8643/healthz": dial tcp 10.134.0.18:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.42:8643/healthz": dial tcp 10.133.0.42:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-8zprn |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.42:8080: connect: connection refused |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-stop-predictor-86dc9fdb65 from 0 to 1 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-stop-v2-predictor-77f778d864 from 0 to 1 | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-stop": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-jb7wd |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-stop-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-5hxnt |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-stop-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-v2-predictor-77f778d864 |
SuccessfulCreate |
Created pod: isvc-canary-stop-v2-predictor-77f778d864-5hxnt | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-predictor-86dc9fdb65 |
SuccessfulCreate |
Created pod: isvc-canary-stop-predictor-86dc9fdb65-jb7wd | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-jb7wd |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-jb7wd |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-5hxnt |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-predictor-86dc9fdb65-jb7wd |
AddedInterface |
Add eth0 [10.133.0.44/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-5hxnt |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-5hxnt |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-v2-predictor-77f778d864-5hxnt |
AddedInterface |
Add eth0 [10.133.0.45/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-jb7wd |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-5hxnt |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-jb7wd |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-jb7wd |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-jb7wd |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-5hxnt |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-5hxnt |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-5hxnt |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-jb7wd |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-jb7wd |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-jb7wd |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-5hxnt |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-5hxnt |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InferenceServiceReady |
InferenceService [isvc-canary-stop] is Ready | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InferenceServiceNotReady |
InferenceService [isvc-canary-stop] is no longer Ready because of: PredictorReady, Ready | |
| (x20) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-jb7wd |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-jb7wd |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-5hxnt |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-5hxnt |
Killing |
Stopping container kube-rbac-proxy | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-5hxnt |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.45:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-jb7wd |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.44:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-jb7wd |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.44:8643/healthz": dial tcp 10.133.0.44:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-5hxnt |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.45:8643/healthz": dial tcp 10.133.0.45:8643: connect: connection refused | |
kserve-ci-e2e-test |
deployment-controller |
isvc-primary-56d0f5-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-primary-56d0f5-predictor-fb478888f from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-56d0f5-predictor-fb478888f-prb6c |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-primary-56d0f5-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-56d0f5 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-56d0f5": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-56d0f5 |
UpdateFailed |
Failed to update status for InferenceService "isvc-primary-56d0f5": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-56d0f5": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-primary-56d0f5-predictor-fb478888f |
SuccessfulCreate |
Created pod: isvc-primary-56d0f5-predictor-fb478888f-prb6c | |
kserve-ci-e2e-test |
multus |
isvc-primary-56d0f5-predictor-fb478888f-prb6c |
AddedInterface |
Add eth0 [10.133.0.46/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-56d0f5-predictor-fb478888f-prb6c |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-56d0f5-predictor-fb478888f-prb6c |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-56d0f5-predictor-fb478888f-prb6c |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-56d0f5-predictor-fb478888f-prb6c |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-56d0f5-predictor-fb478888f-prb6c |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-56d0f5-predictor-fb478888f-prb6c |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-56d0f5-predictor-fb478888f-prb6c |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-56d0f5-predictor-fb478888f-prb6c |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-56d0f5-predictor-fb478888f-prb6c |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-56d0f5-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-56d0f5-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-primary-56d0f5-predictor-fb478888f-prb6c |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.46:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-56d0f5 |
InferenceServiceReady |
InferenceService [isvc-primary-56d0f5] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-56d0f5 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
deployment-controller |
isvc-secondary-56d0f5-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-secondary-56d0f5-predictor-6c98bb5f66 from 0 to 1 | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-56d0f5 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-56d0f5": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
replicaset-controller |
isvc-secondary-56d0f5-predictor-6c98bb5f66 |
SuccessfulCreate |
Created pod: isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-56d0f5 |
UpdateFailed |
Failed to update status for InferenceService "isvc-secondary-56d0f5": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-56d0f5": the object has been modified; please apply your changes to the latest version and try again |
| (x6) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-56d0f5-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-secondary-56d0f5-predictor-serving-cert" not found | |
| (x6) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-56d0f5-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
multus |
isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt |
AddedInterface |
Add eth0 [10.133.0.47/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" already present on machine |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt |
Started |
Started container storage-initializer |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-secondary-56d0f5-predictor-6c98bb5f66-8w4mt_kserve-ci-e2e-test(4bec48ac-c5ac-47ff-a02d-d32fc74da9f4) | |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-56d0f5 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-56d0f5-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-56d0f5-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-56d0f5-predictor-fb478888f-prb6c |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-56d0f5-predictor-fb478888f-prb6c |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-2f3164 |
UpdateFailed |
Failed to update status for InferenceService "isvc-init-fail-2f3164": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-2f3164": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-init-fail-2f3164-predictor-7b67c48cb8 |
SuccessfulCreate |
Created pod: isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-2f3164 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-2f3164": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-init-fail-2f3164-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-init-fail-2f3164-predictor-7b67c48cb8 from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k |
AddedInterface |
Add eth0 [10.133.0.48/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-init-fail-2f3164-predictor-serving-cert" not found | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" already present on machine |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k |
Started |
Started container storage-initializer |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-2f3164 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k |
Created |
Created container: storage-initializer |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-2f3164-predictor-7b67c48cb8-79l5k |
Killing |
Stopping container storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-2483b-predictor-54bfbcdcbd |
SuccessfulCreate |
Created pod: raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj | |
kserve-ci-e2e-test |
multus |
raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj |
AddedInterface |
Add eth0 [10.133.0.49/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-2483b |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-2483b": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-2483b": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-2483b-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-2483b-predictor-54bfbcdcbd from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj |
Started |
Started container kserve-container | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-2483b-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-2483b-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.49:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-2483b |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-2483b |
InferenceServiceReady |
InferenceService [raw-sklearn-2483b] is Ready | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj |
Killing |
Stopping container kube-rbac-proxy | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-2483b-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-2483b-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-2483b-predictor-54bfbcdcbd-kr5kj |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.49:8643/healthz": dial tcp 10.133.0.49:8643: connect: connection refused | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48 |
SuccessfulCreate |
Created pod: raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-aff5e |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-aff5e": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-runtime-aff5e-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "raw-sklearn-runtime-aff5e-predictor-serving-cert" not found | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-aff5e |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-runtime-aff5e": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-aff5e": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b |
AddedInterface |
Add eth0 [10.133.0.50/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-aff5e-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-aff5e-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-aff5e |
InferenceServiceReady |
InferenceService [raw-sklearn-runtime-aff5e] is Ready | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-aff5e |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b |
Killing |
Stopping container kube-rbac-proxy | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-aff5e-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-aff5e-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.50:8643/healthz": dial tcp 10.133.0.50:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-aff5e-predictor-7cb9bfdc48-99r5b |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.50:8080: connect: connection refused |