| Time | Namespace | Component | RelatedObject | Reason | Message |
|---|---|---|---|---|---|
kserve-ci-e2e-test |
isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj to ip-10-0-131-121.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs to ip-10-0-137-14.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-v2-predictor-77f778d864-vgkzc |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vgkzc to ip-10-0-131-121.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-predictor-86dc9fdb65-snddz |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-snddz to ip-10-0-131-121.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-0be34-9dd95d454-bdzdk |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-0be34-9dd95d454-bdzdk to ip-10-0-131-121.ec2.internal | ||
kserve-ci-e2e-test |
isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg to ip-10-0-131-121.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2 to ip-10-0-131-121.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5 to ip-10-0-131-121.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-predictor-cb85cf55c-skqgz |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-skqgz to ip-10-0-137-14.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r to ip-10-0-137-14.ec2.internal | ||
kserve-ci-e2e-test |
isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g to ip-10-0-131-121.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-v2-predictor-86766c4958-lh556 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-lh556 to ip-10-0-137-14.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-v2-predictor-86766c4958-w25q8 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w25q8 to ip-10-0-131-121.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc to ip-10-0-131-121.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9 to ip-10-0-131-121.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j to ip-10-0-131-121.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv to ip-10-0-131-121.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7 to ip-10-0-137-14.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb to ip-10-0-137-14.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4 to ip-10-0-137-14.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd to ip-10-0-131-121.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-predictor-d9759ffd5-7n4r9 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-7n4r9 to ip-10-0-137-14.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69 to ip-10-0-137-14.ec2.internal | ||
kserve-ci-e2e-test |
isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8 to ip-10-0-137-14.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq to ip-10-0-137-14.ec2.internal | ||
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-b15e5 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-b15e5": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-raw-sklearn-batcher-b15e5-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-b15e5 |
UpdateFailed |
Failed to update status for InferenceService "isvc-raw-sklearn-batcher-b15e5": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-b15e5": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8 |
AddedInterface |
Add eth0 [10.132.0.34/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:06e49304d7aff24f41f8f0b7d4ea2dcf886a7d2132c6967087fee7fad06051a4" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564 |
SuccessfulCreate |
Created pod: isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8 | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8 |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8 |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" in 13.621s (13.621s including waiting). Image size: 1555662260 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8 |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-agent@sha256:efaf0bd4be5e59e0fd5df072416f565ae616956e4a23f814ea31d63b5efa6d4b" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8 |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.303s (2.303s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8 |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:efaf0bd4be5e59e0fd5df072416f565ae616956e4a23f814ea31d63b5efa6d4b" in 2.892s (2.892s including waiting). Image size: 240168571 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8 |
Created |
Created container: agent | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-b15e5-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-b15e5-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-b15e5 |
InferenceServiceReady |
InferenceService [isvc-raw-sklearn-batcher-b15e5] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-b15e5 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-b15e5-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8 |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs |
AddedInterface |
Add eth0 [10.132.0.36/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-0be34-predictor-6bf6587864 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4 | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-0be34-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8 |
Killing |
Stopping container kube-rbac-proxy | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-b15e5-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-0be34-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-0be34-predictor-6bf6587864 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-0be34 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-0be34": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-0be34": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-graph-raw-0be34-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:06e49304d7aff24f41f8f0b7d4ea2dcf886a7d2132c6967087fee7fad06051a4" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4 |
AddedInterface |
Add eth0 [10.132.0.35/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:06e49304d7aff24f41f8f0b7d4ea2dcf886a7d2132c6967087fee7fad06051a4" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4 |
Started |
Started container storage-initializer | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.34:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4 |
Started |
Started container kube-rbac-proxy | |
| (x3) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.34:8643/healthz": dial tcp 10.132.0.34:8643: connect: connection refused |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b15e5-predictor-5995f78564-jw8n8 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-0be34-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-0be34-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 17.492s (17.492s including waiting). Image size: 1307355896 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs |
Created |
Created container: kserve-container | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-0be34-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-0be34-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.36:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-0be34 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-0be34] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-0be34 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-0be34 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-0be34 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-0be34] is Ready | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-0be34 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-0be34] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-0be34 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-0be34-9dd95d454 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-0be34-9dd95d454 |
SuccessfulCreate |
Created pod: model-chainer-raw-0be34-9dd95d454-bdzdk | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-0be34-9dd95d454-bdzdk |
AddedInterface |
Add eth0 [10.134.0.25/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-0be34-9dd95d454-bdzdk |
Pulling |
Pulling image "quay.io/opendatahub/kserve-router@sha256:40920ff6a00036cf8932ff07962738516dfac83c34c2855a78a35a4531f74d96" | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-0be34-9dd95d454-bdzdk |
Created |
Created container: model-chainer-raw-0be34 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-0be34-9dd95d454-bdzdk |
Started |
Started container model-chainer-raw-0be34 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-0be34-9dd95d454-bdzdk |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:40920ff6a00036cf8932ff07962738516dfac83c34c2855a78a35a4531f74d96" in 2.449s (2.449s including waiting). Image size: 218226816 bytes. | |
| (x6) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-0be34-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-0be34-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x6) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-0be34-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-0be34-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-c70d2 |
UpdateFailed |
Failed to update status for InferenceService "isvc-xgboost-graph-raw-hpa-c70d2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-graph-raw-hpa-c70d2": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-hpa-c70d2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-0be34-9dd95d454-bdzdk |
Killing |
Stopping container model-chainer-raw-0be34 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-c70d2 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-hpa-c70d2": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-hpa-c70d2": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-hpa-c70d2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:06e49304d7aff24f41f8f0b7d4ea2dcf886a7d2132c6967087fee7fad06051a4" | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-graph-raw-hpa-c70d2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2 |
AddedInterface |
Add eth0 [10.134.0.26/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq |
AddedInterface |
Add eth0 [10.132.0.37/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:06e49304d7aff24f41f8f0b7d4ea2dcf886a7d2132c6967087fee7fad06051a4" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.35:8643/healthz": dial tcp 10.132.0.35:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:06e49304d7aff24f41f8f0b7d4ea2dcf886a7d2132c6967087fee7fad06051a4" in 3.035s (3.035s including waiting). Image size: 300978647 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-0be34-predictor-74bb5dbfc-2r9vs |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.36:8643/healthz": dial tcp 10.132.0.36:8643: connect: connection refused | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-0be34-predictor-6bf6587864-vnkm4 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.35:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2 |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2 |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2 |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 17.976s (17.976s including waiting). Image size: 1307355896 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2 |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.256s (2.256s including waiting). Image size: 211946088 bytes. | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-0be34-9dd95d454-bdzdk |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-c70d2-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-c70d2-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-c70d2-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-c70d2-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.37:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-c70d2 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-c70d2 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-hpa-c70d2] is Ready | |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.26:8080: connect: connection refused |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-c70d2 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-c70d2 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-hpa-c70d2] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-hpa-c70d2 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-hpa-c70d2-5f8cf6db8b from 0 to 1 | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-c70d2 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-hpa-c70d2] is Ready | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-hpa-c70d2-5f8cf6db8b |
SuccessfulCreate |
Created pod: model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9 |
Started |
Started container model-chainer-raw-hpa-c70d2 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9 |
Created |
Created container: model-chainer-raw-hpa-c70d2 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9 |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:40920ff6a00036cf8932ff07962738516dfac83c34c2855a78a35a4531f74d96" already present on machine | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9 |
AddedInterface |
Add eth0 [10.134.0.27/23] from ovn-kubernetes | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-c70d2-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-c70d2-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-c70d2-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-c70d2-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9 |
Killing |
Stopping container model-chainer-raw-hpa-c70d2 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "message-dumper-raw-22083-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-22083 |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-22083": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-22083": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c70d2-predictor-7c7cb8b49-8qmv2 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-22083-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-22083-predictor-7fc855cfcc from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-22083-predictor-7fc855cfcc |
SuccessfulCreate |
Created pod: message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c70d2-predictor-789bb95dd8-cvqlq |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.37:8643/healthz": dial tcp 10.132.0.37:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd |
Pulling |
Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd |
AddedInterface |
Add eth0 [10.134.0.28/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd |
Pulled |
Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.094s (1.094s including waiting). Image size: 14813193 bytes. | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-22083 |
InferenceServiceReady |
InferenceService [message-dumper-raw-22083] is Ready | |
| (x8) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-22083 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-logger-raw-22083-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-22083 |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-22083": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-22083": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-22083-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-22083-predictor-69fcbb7548 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-22083-predictor-69fcbb7548 |
SuccessfulCreate |
Created pod: isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r |
AddedInterface |
Add eth0 [10.132.0.38/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:06e49304d7aff24f41f8f0b7d4ea2dcf886a7d2132c6967087fee7fad06051a4" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:efaf0bd4be5e59e0fd5df072416f565ae616956e4a23f814ea31d63b5efa6d4b" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r |
Started |
Started container agent | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-c70d2-5f8cf6db8b-wgdr9 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-22083-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-22083-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-22083-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-22083-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-22083 |
InferenceServiceReady |
InferenceService [isvc-logger-raw-22083] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-22083 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-22083-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-22083-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.38:8080: connect: connection refused |
kserve-ci-e2e-test |
multus |
isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69 |
AddedInterface |
Add eth0 [10.132.0.39/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-scale-raw-4c735-predictor-7478ccd994 |
SuccessfulCreate |
Created pod: isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-scale-raw-4c735-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-scale-raw-4c735-predictor-7478ccd994 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:06e49304d7aff24f41f8f0b7d4ea2dcf886a7d2132c6967087fee7fad06051a4" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-4c735 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-scale-raw-4c735": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scale-raw-4c735": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-22083-predictor-7fc855cfcc-kv9rd |
Killing |
Stopping container kube-rbac-proxy | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69 |
Created |
Created container: kube-rbac-proxy | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-22083-predictor-69fcbb7548-s8k2r |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.38:8643/healthz": dial tcp 10.132.0.38:8643: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-4c735-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-4c735-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x8) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-4c735-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x8) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-4c735-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-4c735 |
InferenceServiceReady |
InferenceService [isvc-sklearn-scale-raw-4c735] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-4c735 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-predictor-cb85cf55c |
SuccessfulCreate |
Created pod: isvc-canary-create-predictor-cb85cf55c-skqgz | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-v2-predictor-6d8d4768d4 |
SuccessfulCreate |
Created pod: isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-create-v2-predictor-6d8d4768d4 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-create": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-create-predictor-cb85cf55c from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7 |
AddedInterface |
Add eth0 [10.132.0.41/23] from ovn-kubernetes | |
| (x18) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.39:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:06e49304d7aff24f41f8f0b7d4ea2dcf886a7d2132c6967087fee7fad06051a4" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7 |
Started |
Started container storage-initializer | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-4c735-predictor-7478ccd994-v8s69 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.39:8643/healthz": dial tcp 10.132.0.39:8643: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-skqgz |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-skqgz |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-skqgz |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:06e49304d7aff24f41f8f0b7d4ea2dcf886a7d2132c6967087fee7fad06051a4" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-predictor-cb85cf55c-skqgz |
AddedInterface |
Add eth0 [10.132.0.40/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-skqgz |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-skqgz |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-skqgz |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-skqgz |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-skqgz |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-skqgz |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
InferenceServiceReady |
InferenceService [isvc-canary-create] is Ready | |
| (x15) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-skqgz |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-skqgz |
Killing |
Stopping container kserve-container | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.41:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-6d8d4768d4-8qfl7 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.41:8643/healthz": dial tcp 10.132.0.41:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-skqgz |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.40:8643/healthz": dial tcp 10.132.0.40:8643: connect: connection refused | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-cb85cf55c-skqgz |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.40:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-promote": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-lh556 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-promote-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-predictor-d9759ffd5-7n4r9 |
AddedInterface |
Add eth0 [10.132.0.42/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-v2-predictor-86766c4958 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-7n4r9 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-7n4r9 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:06e49304d7aff24f41f8f0b7d4ea2dcf886a7d2132c6967087fee7fad06051a4" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-7n4r9 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-predictor-d9759ffd5 |
SuccessfulCreate |
Created pod: isvc-canary-promote-predictor-d9759ffd5-7n4r9 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-predictor-d9759ffd5 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-v2-predictor-86766c4958 |
SuccessfulCreate |
Created pod: isvc-canary-promote-v2-predictor-86766c4958-lh556 | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-v2-predictor-86766c4958-lh556 |
AddedInterface |
Add eth0 [10.132.0.43/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-lh556 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:06e49304d7aff24f41f8f0b7d4ea2dcf886a7d2132c6967087fee7fad06051a4" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-lh556 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-lh556 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-7n4r9 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-7n4r9 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-7n4r9 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-7n4r9 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-7n4r9 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-7n4r9 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-lh556 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-lh556 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-lh556 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-lh556 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-lh556 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-lh556 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InferenceServiceNotReady |
InferenceService [isvc-canary-promote] is no longer Ready because of: PredictorReady, Ready, Stopped | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-w25q8 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:06e49304d7aff24f41f8f0b7d4ea2dcf886a7d2132c6967087fee7fad06051a4" already present on machine | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-v2-predictor-86766c4958 |
SuccessfulCreate |
Created pod: isvc-canary-promote-v2-predictor-86766c4958-w25q8 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-7n4r9 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-w25q8 |
Started |
Started container storage-initializer | |
| (x22) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-v2-predictor-86766c4958 from 1 to 2 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-7n4r9 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-v2-predictor-86766c4958-w25q8 |
AddedInterface |
Add eth0 [10.134.0.29/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-w25q8 |
Created |
Created container: storage-initializer | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-7n4r9 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.42:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-d9759ffd5-7n4r9 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.42:8643/healthz": dial tcp 10.132.0.42:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-w25q8 |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-w25q8 |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" in 13.259s (13.259s including waiting). Image size: 1555662260 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-w25q8 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-w25q8 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-w25q8 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-w25q8 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-w25q8 |
Started |
Started container kube-rbac-proxy | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InferenceServiceReady |
InferenceService [isvc-canary-promote] is Ready |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-w25q8 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-lh556 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-lh556 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-w25q8 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-w25q8 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.29:8643/healthz": dial tcp 10.134.0.29:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-w25q8 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.29:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-lh556 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.43:8643/healthz": dial tcp 10.132.0.43:8643: connect: connection refused | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-86766c4958-lh556 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.43:8080: connect: connection refused |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-5bbb8c8cfc |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-v2-predictor-84f48b7656 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-v2-predictor-84f48b7656 |
SuccessfulCreate |
Created pod: isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-predictor-5bbb8c8cfc from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5 |
AddedInterface |
Add eth0 [10.134.0.31/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc |
AddedInterface |
Add eth0 [10.134.0.30/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:06e49304d7aff24f41f8f0b7d4ea2dcf886a7d2132c6967087fee7fad06051a4" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:06e49304d7aff24f41f8f0b7d4ea2dcf886a7d2132c6967087fee7fad06051a4" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc |
Created |
Created container: kserve-container | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.31:8080: connect: connection refused |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-5bbb8c8cfc |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InferenceServiceNotReady |
InferenceService [isvc-canary-rollback] is no longer Ready because of: PredictorReady, Ready, Stopped | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-rollback": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again |
| (x20) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-predictor-5bbb8c8cfc from 1 to 2 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb |
AddedInterface |
Add eth0 [10.132.0.44/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:06e49304d7aff24f41f8f0b7d4ea2dcf886a7d2132c6967087fee7fad06051a4" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-84f48b7656-h2tj5 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.31:8643/healthz": dial tcp 10.134.0.31:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InferenceServiceReady |
InferenceService [isvc-canary-rollback] is Ready |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.44:8643/healthz": dial tcp 10.132.0.44:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.30:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-qnljc |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.30:8643/healthz": dial tcp 10.134.0.30:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-5bbb8c8cfc-47hvb |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.44:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-snddz |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-vgkzc |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-v2-predictor-77f778d864 |
SuccessfulCreate |
Created pod: isvc-canary-stop-v2-predictor-77f778d864-vgkzc | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-stop-v2-predictor-77f778d864 from 0 to 1 | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-stop": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-vgkzc |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-vgkzc |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:06e49304d7aff24f41f8f0b7d4ea2dcf886a7d2132c6967087fee7fad06051a4" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-v2-predictor-77f778d864-vgkzc |
AddedInterface |
Add eth0 [10.134.0.33/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-stop-predictor-86dc9fdb65 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-predictor-86dc9fdb65 |
SuccessfulCreate |
Created pod: isvc-canary-stop-predictor-86dc9fdb65-snddz | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-snddz |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-snddz |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:06e49304d7aff24f41f8f0b7d4ea2dcf886a7d2132c6967087fee7fad06051a4" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-predictor-86dc9fdb65-snddz |
AddedInterface |
Add eth0 [10.134.0.32/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-snddz |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-vgkzc |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-snddz |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-snddz |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-snddz |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-snddz |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-snddz |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-vgkzc |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-vgkzc |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-vgkzc |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-vgkzc |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-vgkzc |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InferenceServiceReady |
InferenceService [isvc-canary-stop] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-snddz |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-vgkzc |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-vgkzc |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InferenceServiceNotReady |
InferenceService [isvc-canary-stop] is no longer Ready because of: PredictorReady, Ready | |
| (x18) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-snddz |
Killing |
Stopping container kube-rbac-proxy | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-vgkzc |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.33:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-77f778d864-vgkzc |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.33:8643/healthz": dial tcp 10.134.0.33:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-snddz |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.32:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-86dc9fdb65-snddz |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.32:8643/healthz": dial tcp 10.134.0.32:8643: connect: connection refused | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-9bf0ca |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-9bf0ca": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-primary-9bf0ca-predictor-7969d99c69 |
SuccessfulCreate |
Created pod: isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g | |
kserve-ci-e2e-test |
deployment-controller |
isvc-primary-9bf0ca-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-primary-9bf0ca-predictor-7969d99c69 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-9bf0ca |
UpdateFailed |
Failed to update status for InferenceService "isvc-primary-9bf0ca": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-9bf0ca": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g |
AddedInterface |
Add eth0 [10.134.0.34/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:06e49304d7aff24f41f8f0b7d4ea2dcf886a7d2132c6967087fee7fad06051a4" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g |
Created |
Created container: kube-rbac-proxy | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-9bf0ca-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-9bf0ca-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-9bf0ca |
InferenceServiceReady |
InferenceService [isvc-primary-9bf0ca] is Ready | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-9bf0ca |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
replicaset-controller |
isvc-secondary-9bf0ca-predictor-7bf7b679b6 |
SuccessfulCreate |
Created pod: isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-9bf0ca |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-9bf0ca": the object has been modified; please apply your changes to the latest version and try again |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-9bf0ca-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-9bf0ca-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-9bf0ca |
UpdateFailed |
Failed to update status for InferenceService "isvc-secondary-9bf0ca": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-9bf0ca": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
deployment-controller |
isvc-secondary-9bf0ca-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-secondary-9bf0ca-predictor-7bf7b679b6 from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj |
AddedInterface |
Add eth0 [10.134.0.35/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:06e49304d7aff24f41f8f0b7d4ea2dcf886a7d2132c6967087fee7fad06051a4" already present on machine |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj |
Started |
Started container storage-initializer |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-secondary-9bf0ca-predictor-7bf7b679b6-9gpwj_kserve-ci-e2e-test(dbc9e2a9-e2c3-4845-b8dd-9df66bdd96e7) | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-9bf0ca |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-9bf0ca-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-9bf0ca-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.34:8643/healthz": dial tcp 10.134.0.34:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-primary-9bf0ca-predictor-7969d99c69-qmc4g |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.34:8080: connect: connection refused |
kserve-ci-e2e-test |
replicaset-controller |
isvc-init-fail-6a365a-predictor-5467b89cf6 |
SuccessfulCreate |
Created pod: isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg | |
kserve-ci-e2e-test |
deployment-controller |
isvc-init-fail-6a365a-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-init-fail-6a365a-predictor-5467b89cf6 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-6a365a |
UpdateFailed |
Failed to update status for InferenceService "isvc-init-fail-6a365a": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-6a365a": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-6a365a |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-6a365a": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg |
AddedInterface |
Add eth0 [10.134.0.36/23] from ovn-kubernetes | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-6a365a |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:06e49304d7aff24f41f8f0b7d4ea2dcf886a7d2132c6967087fee7fad06051a4" already present on machine |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-6a365a-predictor-5467b89cf6-8hvpg |
Killing |
Stopping container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "raw-sklearn-fcfc1-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-fcfc1-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-fcfc1-predictor-f6dd84fdd from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-fcfc1-predictor-f6dd84fdd |
SuccessfulCreate |
Created pod: raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-fcfc1 |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-fcfc1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-fcfc1": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j |
AddedInterface |
Add eth0 [10.134.0.37/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:06e49304d7aff24f41f8f0b7d4ea2dcf886a7d2132c6967087fee7fad06051a4" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-fcfc1-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-fcfc1-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-fcfc1 |
InferenceServiceReady |
InferenceService [raw-sklearn-fcfc1] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-fcfc1 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-fcfc1-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-fcfc1-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.37:8643/healthz": dial tcp 10.134.0.37:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-fcfc1-predictor-f6dd84fdd-m492j |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.37:8080: connect: connection refused |
kserve-ci-e2e-test |
multus |
raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv |
AddedInterface |
Add eth0 [10.134.0.38/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-8771a |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-runtime-8771a": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-8771a": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-runtime-8771a-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-runtime-8771a-predictor-85dbf9b964 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-runtime-8771a-predictor-85dbf9b964 |
SuccessfulCreate |
Created pod: raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:06e49304d7aff24f41f8f0b7d4ea2dcf886a7d2132c6967087fee7fad06051a4" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv |
Created |
Created container: kserve-container | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-8771a-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-8771a-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-8771a |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-8771a |
InferenceServiceReady |
InferenceService [raw-sklearn-runtime-8771a] is Ready | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv |
Killing |
Stopping container kserve-container | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-8771a-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-8771a-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.38:8643/healthz": dial tcp 10.134.0.38:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-8771a-predictor-85dbf9b964-sjwqv |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.38:8080: connect: connection refused |