| Time | Namespace | Component | RelatedObject | Reason | Message |
|---|---|---|---|---|---|
kserve-ci-e2e-test |
raw-sklearn-fb75d-predictor-768dc4cffb-4phk2 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-fb75d-predictor-768dc4cffb-4phk2 to ip-10-0-138-17.ec2.internal | ||
kserve-ci-e2e-test |
isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf to ip-10-0-138-17.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-b8148-predictor-86f78f7649-2xqrx |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-b8148-predictor-86f78f7649-2xqrx to ip-10-0-138-17.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-predictor-5bf5667989-dv7nc |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-dv7nc to ip-10-0-138-17.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d to ip-10-0-138-17.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-predictor-594f5f9bd5-krwrj |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-krwrj to ip-10-0-138-17.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf to ip-10-0-138-17.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf to ip-10-0-138-17.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-5fb11-7f4bdffbb9-2zv2q |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-5fb11-7f4bdffbb9-2zv2q to ip-10-0-128-156.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-b4645bb5c-bj66w |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-bj66w to ip-10-0-139-137.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-hpa-c3acd-6ff77f57cd-s6h2q |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-hpa-c3acd-6ff77f57cd-s6h2q to ip-10-0-128-156.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7 to ip-10-0-138-17.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b to ip-10-0-138-17.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r to ip-10-0-138-17.ec2.internal | ||
kserve-ci-e2e-test |
isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s to ip-10-0-138-17.ec2.internal | ||
kserve-ci-e2e-test |
isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl to ip-10-0-138-17.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4 to ip-10-0-138-17.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-v2-predictor-5b976d889b-7pg8g |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-7pg8g to ip-10-0-138-17.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-b4645bb5c-4b62h |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-4b62h to ip-10-0-138-17.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-v2-predictor-f69c9c758-hgz9z |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-hgz9z to ip-10-0-138-17.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-v2-predictor-f69c9c758-95bkm |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-95bkm to ip-10-0-139-137.ec2.internal | ||
kserve-ci-e2e-test |
isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn to ip-10-0-138-17.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx to ip-10-0-138-17.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-predictor-5d7dcd97d5-c87qb |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-c87qb to ip-10-0-138-17.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt to ip-10-0-138-17.ec2.internal | ||
kserve-ci-e2e-test |
deployment-controller |
isvc-raw-sklearn-batcher-2ecea-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-2ecea |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-2ecea": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s |
AddedInterface |
Add eth0 [10.132.0.32/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-2ecea |
UpdateFailed |
Failed to update status for InferenceService "isvc-raw-sklearn-batcher-2ecea": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-2ecea": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd |
SuccessfulCreate |
Created pod: isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" in 3.374s (3.374s including waiting). Image size: 300977080 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821" in 13.772s (13.772s including waiting). Image size: 1555722147 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.333s (2.333s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s |
Pulling |
Pulling image "quay.io/opendatahub/kserve-agent@sha256:d2cb06ca9eeb023462e079ca78f69f867aac7089d74cc32b24351cfb0f653480" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:d2cb06ca9eeb023462e079ca78f69f867aac7089d74cc32b24351cfb0f653480" in 2.636s (2.636s including waiting). Image size: 240170074 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s |
Started |
Started container agent | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-2ecea-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-2ecea-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-2ecea-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-2ecea-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-2ecea |
InferenceServiceReady |
InferenceService [isvc-raw-sklearn-batcher-2ecea] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-2ecea |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-5fb11-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-graph-raw-5fb11-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-5fb11 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-5fb11": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-5fb11": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-5fb11-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt |
AddedInterface |
Add eth0 [10.132.0.34/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r |
AddedInterface |
Add eth0 [10.132.0.33/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r |
Created |
Created container: kserve-container | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.32:8080: connect: connection refused |
| (x3) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-2ecea-predictor-59974dcdbd-gmg4s |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.32:8643/healthz": dial tcp 10.132.0.32:8643: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 18.45s (18.45s including waiting). Image size: 1307355896 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt |
Started |
Started container kserve-container | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-5fb11-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-5fb11-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-5fb11-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-5fb11-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-5fb11 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-5fb11 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-5fb11] is Ready | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-5fb11 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-5fb11] is Ready | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-5fb11 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-5fb11 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-5fb11] is Ready | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-5fb11-7f4bdffbb9-2zv2q |
Pulling |
Pulling image "quay.io/opendatahub/kserve-router@sha256:2d78b1e75ce4a3e633713c1f964bde1c2425164638b26ad77af20d6340c844d7" | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-5fb11-7f4bdffbb9-2zv2q |
AddedInterface |
Add eth0 [10.133.0.28/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-5fb11 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-5fb11-7f4bdffbb9 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-5fb11-7f4bdffbb9 |
SuccessfulCreate |
Created pod: model-chainer-raw-5fb11-7f4bdffbb9-2zv2q | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-5fb11-7f4bdffbb9-2zv2q |
Created |
Created container: model-chainer-raw-5fb11 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-5fb11-7f4bdffbb9-2zv2q |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:2d78b1e75ce4a3e633713c1f964bde1c2425164638b26ad77af20d6340c844d7" in 1.785s (1.785s including waiting). Image size: 218226779 bytes. | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-5fb11-7f4bdffbb9-2zv2q |
Started |
Started container model-chainer-raw-5fb11 | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-5fb11-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-5fb11-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-5fb11-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-5fb11-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-5fb11-7f4bdffbb9-2zv2q |
Killing |
Stopping container model-chainer-raw-5fb11 | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b |
AddedInterface |
Add eth0 [10.132.0.35/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-graph-raw-hpa-c3acd-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-hpa-c3acd-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-hpa-c3acd-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.34:8080: connect: connection refused |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf |
AddedInterface |
Add eth0 [10.132.0.36/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.33:8643/healthz": dial tcp 10.132.0.33:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-5fb11-predictor-854c67d48d-lp94r |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.33:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf |
Pulled |
Container image "kserve/xgbserver:latest" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821" already present on machine | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-5fb11-predictor-5cf465cc65-p7zjt |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.34:8643/healthz": dial tcp 10.132.0.34:8643: connect: connection refused |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-5fb11-7f4bdffbb9-2zv2q |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-c3acd-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-c3acd-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-c3acd-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-c3acd-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-c3acd |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-hpa-c3acd] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-c3acd |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-c3acd |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-c3acd |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-hpa-c3acd] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-hpa-c3acd |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-hpa-c3acd-6ff77f57cd from 0 to 1 | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-hpa-c3acd-6ff77f57cd-s6h2q |
AddedInterface |
Add eth0 [10.133.0.29/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-c3acd |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-hpa-c3acd] is Ready | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-hpa-c3acd-6ff77f57cd |
SuccessfulCreate |
Created pod: model-chainer-raw-hpa-c3acd-6ff77f57cd-s6h2q | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-c3acd-6ff77f57cd-s6h2q |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:2d78b1e75ce4a3e633713c1f964bde1c2425164638b26ad77af20d6340c844d7" already present on machine | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-c3acd-6ff77f57cd-s6h2q |
Started |
Started container model-chainer-raw-hpa-c3acd | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-c3acd-6ff77f57cd-s6h2q |
Created |
Created container: model-chainer-raw-hpa-c3acd | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-c3acd-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-c3acd-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-c3acd-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-c3acd-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-b8148-predictor-86f78f7649-2xqrx |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "message-dumper-raw-b8148-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-c3acd-6ff77f57cd-s6h2q |
Killing |
Stopping container model-chainer-raw-hpa-c3acd | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-b8148 |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-b8148": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-b8148": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-b8148-predictor-86f78f7649 |
SuccessfulCreate |
Created pod: message-dumper-raw-b8148-predictor-86f78f7649-2xqrx | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-b8148-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-b8148-predictor-86f78f7649 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-b8148-predictor-86f78f7649-2xqrx |
Pulling |
Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-b8148-predictor-86f78f7649-2xqrx |
AddedInterface |
Add eth0 [10.132.0.37/23] from ovn-kubernetes | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.35:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-b8148-predictor-86f78f7649-2xqrx |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.36:8643/healthz": dial tcp 10.132.0.36:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-b8148-predictor-86f78f7649-2xqrx |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-b8148-predictor-86f78f7649-2xqrx |
Pulled |
Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.135s (1.135s including waiting). Image size: 14813193 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-c3acd-predictor-7776b474fb-fbg7b |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.35:8643/healthz": dial tcp 10.132.0.35:8643: connect: connection refused | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-c3acd-predictor-6dd56b77b9-ktljf |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.36:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-b8148-predictor-86f78f7649-2xqrx |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-b8148-predictor-86f78f7649-2xqrx |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-b8148-predictor-86f78f7649-2xqrx |
Started |
Started container kube-rbac-proxy | |
| (x7) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-b8148 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-b8148 |
InferenceServiceReady |
InferenceService [message-dumper-raw-b8148] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-b8148-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-b8148-predictor-56cd48cdb4 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-b8148-predictor-56cd48cdb4 |
SuccessfulCreate |
Created pod: isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-b8148 |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-b8148": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-b8148": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7 |
AddedInterface |
Add eth0 [10.132.0.38/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7 |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7 |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7 |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:d2cb06ca9eeb023462e079ca78f69f867aac7089d74cc32b24351cfb0f653480" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7 |
Started |
Started container kube-rbac-proxy | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-c3acd-6ff77f57cd-s6h2q |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-b8148-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-b8148-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-b8148-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-b8148-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-b8148 |
InferenceServiceReady |
InferenceService [isvc-logger-raw-b8148] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-b8148 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-b8148-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-b8148-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-scale-raw-e2d9f-predictor-serving-cert" not found | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-e2d9f |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-scale-raw-e2d9f": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scale-raw-e2d9f": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-b8148-predictor-86f78f7649-2xqrx |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b |
SuccessfulCreate |
Created pod: isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-scale-raw-e2d9f-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7 |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-e2d9f |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scale-raw-e2d9f": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-b8148-predictor-86f78f7649-2xqrx |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4 |
AddedInterface |
Add eth0 [10.132.0.39/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4 |
Started |
Started container storage-initializer | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4 |
Created |
Created container: kserve-container | |
| (x3) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.38:8643/healthz": dial tcp 10.132.0.38:8643: connect: connection refused |
| (x12) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8148-predictor-56cd48cdb4-xjmb7 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.38:8080: connect: connection refused |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-e2d9f-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-e2d9f-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-e2d9f |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-e2d9f |
InferenceServiceReady |
InferenceService [isvc-sklearn-scale-raw-e2d9f] is Ready | |
| (x10) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-e2d9f-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x10) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-e2d9f-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4 |
Killing |
Stopping container kserve-container | |
| (x19) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.39:8080: connect: connection refused |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-v2-predictor-5b976d889b |
SuccessfulCreate |
Created pod: isvc-canary-create-v2-predictor-5b976d889b-7pg8g | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-create-v2-predictor-5b976d889b from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-predictor-594f5f9bd5 |
SuccessfulCreate |
Created pod: isvc-canary-create-predictor-594f5f9bd5-krwrj | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-create-predictor-594f5f9bd5 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5b976d889b-7pg8g |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-create-v2-predictor-serving-cert" not found | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-create": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-594f5f9bd5-krwrj |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-594f5f9bd5-krwrj |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5b976d889b-7pg8g |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5b976d889b-7pg8g |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-v2-predictor-5b976d889b-7pg8g |
AddedInterface |
Add eth0 [10.132.0.41/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-594f5f9bd5-krwrj |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5b976d889b-7pg8g |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-predictor-594f5f9bd5-krwrj |
AddedInterface |
Add eth0 [10.132.0.40/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e2d9f-predictor-6fd8fd9b8b-lptl4 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.39:8643/healthz": dial tcp 10.132.0.39:8643: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-594f5f9bd5-krwrj |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-594f5f9bd5-krwrj |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-594f5f9bd5-krwrj |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-594f5f9bd5-krwrj |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-594f5f9bd5-krwrj |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-594f5f9bd5-krwrj |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5b976d889b-7pg8g |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5b976d889b-7pg8g |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5b976d889b-7pg8g |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5b976d889b-7pg8g |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5b976d889b-7pg8g |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5b976d889b-7pg8g |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821" already present on machine | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
InferenceServiceReady |
InferenceService [isvc-canary-create] is Ready | |
| (x14) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-594f5f9bd5-krwrj |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-594f5f9bd5-krwrj |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5b976d889b-7pg8g |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5b976d889b-7pg8g |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5b976d889b-7pg8g |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.41:8643/healthz": dial tcp 10.132.0.41:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5b976d889b-7pg8g |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.41:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-594f5f9bd5-krwrj |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.40:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-594f5f9bd5-krwrj |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.40:8643/healthz": dial tcp 10.132.0.40:8643: connect: connection refused | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-v2-predictor-f69c9c758 |
SuccessfulCreate |
Created pod: isvc-canary-promote-v2-predictor-f69c9c758-hgz9z | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-v2-predictor-f69c9c758 from 0 to 1 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-predictor-5d7dcd97d5 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-predictor-5d7dcd97d5 |
SuccessfulCreate |
Created pod: isvc-canary-promote-predictor-5d7dcd97d5-c87qb | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-hgz9z |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-hgz9z |
Started |
Started container storage-initializer | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-promote": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-hgz9z |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-v2-predictor-f69c9c758-hgz9z |
AddedInterface |
Add eth0 [10.132.0.43/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5d7dcd97d5-c87qb |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-promote-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-predictor-5d7dcd97d5-c87qb |
AddedInterface |
Add eth0 [10.132.0.42/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5d7dcd97d5-c87qb |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5d7dcd97d5-c87qb |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5d7dcd97d5-c87qb |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5d7dcd97d5-c87qb |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5d7dcd97d5-c87qb |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5d7dcd97d5-c87qb |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5d7dcd97d5-c87qb |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-hgz9z |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-hgz9z |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-hgz9z |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-hgz9z |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5d7dcd97d5-c87qb |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-hgz9z |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-hgz9z |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5d7dcd97d5-c87qb |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-v2-predictor-f69c9c758 from 1 to 2 | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-v2-predictor-f69c9c758-95bkm |
AddedInterface |
Add eth0 [10.134.0.14/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-v2-predictor-f69c9c758 |
SuccessfulCreate |
Created pod: isvc-canary-promote-v2-predictor-f69c9c758-95bkm | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-95bkm |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-95bkm |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-95bkm |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InferenceServiceNotReady |
InferenceService [isvc-canary-promote] is no longer Ready because of: PredictorReady, Ready, Stopped | |
| (x20) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5d7dcd97d5-c87qb |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5d7dcd97d5-c87qb |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5d7dcd97d5-c87qb |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.42:8643/healthz": dial tcp 10.132.0.42:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5d7dcd97d5-c87qb |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.42:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-95bkm |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-95bkm |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821" in 12.83s (12.83s including waiting). Image size: 1555722147 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-95bkm |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-95bkm |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-95bkm |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-95bkm |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-95bkm |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-95bkm |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.28s (2.28s including waiting). Image size: 211946088 bytes. | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-95bkm |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.14:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InferenceServiceReady |
InferenceService [isvc-canary-promote] is Ready |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-95bkm |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-hgz9z |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-hgz9z |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-95bkm |
Killing |
Stopping container kserve-container | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-hgz9z |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.43:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-hgz9z |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.43:8643/healthz": dial tcp 10.132.0.43:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-f69c9c758-95bkm |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.14:8643/healthz": dial tcp 10.134.0.14:8643: connect: connection refused | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-predictor-b4645bb5c from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-b4645bb5c |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-b4645bb5c-4b62h | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-v2-predictor-648cbdfbd8 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-v2-predictor-648cbdfbd8 |
SuccessfulCreate |
Created pod: isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-4b62h |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-4b62h |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-b4645bb5c-4b62h |
AddedInterface |
Add eth0 [10.132.0.44/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-4b62h |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d |
AddedInterface |
Add eth0 [10.132.0.45/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-4b62h |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-4b62h |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-4b62h |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-4b62h |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-4b62h |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-4b62h |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d |
Created |
Created container: kserve-container | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.45:8080: connect: connection refused |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-predictor-b4645bb5c from 1 to 2 | |
| (x19) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x3) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-rollback": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-b4645bb5c |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-b4645bb5c-bj66w | |
| (x3) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-bj66w |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-bj66w |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-bj66w |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-b4645bb5c-bj66w |
AddedInterface |
Add eth0 [10.134.0.15/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-648cbdfbd8-xjq9d |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.45:8643/healthz": dial tcp 10.132.0.45:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-bj66w |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-bj66w |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-bj66w |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-bj66w |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-bj66w |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-bj66w |
Started |
Started container kube-rbac-proxy | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-bj66w |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.15:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InferenceServiceReady |
InferenceService [isvc-canary-rollback] is Ready |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-4b62h |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-4b62h |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-bj66w |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-bj66w |
Killing |
Stopping container kube-rbac-proxy | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-4b62h |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.44:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-4b62h |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.44:8643/healthz": dial tcp 10.132.0.44:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-b4645bb5c-bj66w |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.15:8643/healthz": dial tcp 10.134.0.15:8643: connect: connection refused | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-stop": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-stop-v2-predictor-6c87f69f44 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-v2-predictor-6c87f69f44 |
SuccessfulCreate |
Created pod: isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-predictor-5bf5667989 |
SuccessfulCreate |
Created pod: isvc-canary-stop-predictor-5bf5667989-dv7nc | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-stop-predictor-5bf5667989 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf |
AddedInterface |
Add eth0 [10.132.0.47/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5bf5667989-dv7nc |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-stop-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-predictor-5bf5667989-dv7nc |
AddedInterface |
Add eth0 [10.132.0.46/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5bf5667989-dv7nc |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5bf5667989-dv7nc |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5bf5667989-dv7nc |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5bf5667989-dv7nc |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5bf5667989-dv7nc |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5bf5667989-dv7nc |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5bf5667989-dv7nc |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5bf5667989-dv7nc |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5bf5667989-dv7nc |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821" already present on machine | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InferenceServiceReady |
InferenceService [isvc-canary-stop] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5bf5667989-dv7nc |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5bf5667989-dv7nc |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InferenceServiceNotReady |
InferenceService [isvc-canary-stop] is no longer Ready because of: PredictorReady, Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf |
Killing |
Stopping container kube-rbac-proxy | |
| (x19) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.47:8643/healthz": dial tcp 10.132.0.47:8643: connect: connection refused | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-6c87f69f44-rsxgf |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.47:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5bf5667989-dv7nc |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.46:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-5bf5667989-dv7nc |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.46:8643/healthz": dial tcp 10.132.0.46:8643: connect: connection refused | |
kserve-ci-e2e-test |
multus |
isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl |
AddedInterface |
Add eth0 [10.132.0.48/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
deployment-controller |
isvc-primary-82c139-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-primary-82c139-predictor-7cdf8bbf6f from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-primary-82c139-predictor-7cdf8bbf6f |
SuccessfulCreate |
Created pod: isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-82c139 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-82c139": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-82c139 |
UpdateFailed |
Failed to update status for InferenceService "isvc-primary-82c139": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-82c139": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl |
Created |
Created container: kube-rbac-proxy | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-82c139-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-82c139-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.48:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-82c139 |
InferenceServiceReady |
InferenceService [isvc-primary-82c139] is Ready | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-82c139 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
multus |
isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf |
AddedInterface |
Add eth0 [10.132.0.49/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-82c139 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-82c139": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
deployment-controller |
isvc-secondary-82c139-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-secondary-82c139-predictor-9d5ddc4d6 from 0 to 1 | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-82c139-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-82c139-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-82c139 |
UpdateFailed |
Failed to update status for InferenceService "isvc-secondary-82c139": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-82c139": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
replicaset-controller |
isvc-secondary-82c139-predictor-9d5ddc4d6 |
SuccessfulCreate |
Created pod: isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf |
Created |
Created container: storage-initializer |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-secondary-82c139-predictor-9d5ddc4d6-jljqf_kserve-ci-e2e-test(afc8dbd3-51e8-42e8-b1d1-f1b8841be411) | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-82c139 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-82c139-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-82c139-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-82c139-predictor-7cdf8bbf6f-mfxrl |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-fbe44c |
UpdateFailed |
Failed to update status for InferenceService "isvc-init-fail-fbe44c": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-fbe44c": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-init-fail-fbe44c-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-init-fail-fbe44c-predictor-55dc47c94d from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-init-fail-fbe44c-predictor-55dc47c94d |
SuccessfulCreate |
Created pod: isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-fbe44c |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-fbe44c": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn |
AddedInterface |
Add eth0 [10.132.0.50/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn |
Started |
Started container storage-initializer |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-fbe44c |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn |
FailedToRetrieveImagePullSecret |
Unable to retrieve some image pull secrets (fail-s3-sa-fbe44c-dockercfg-s86lp); attempting to pull the image may not succeed. | |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn |
Killing |
Stopping container storage-initializer | |
| (x3) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-fbe44c-predictor-55dc47c94d-ln8sn |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-init-fail-fbe44c-predictor-serving-cert" not found |
kserve-ci-e2e-test |
multus |
raw-sklearn-fb75d-predictor-768dc4cffb-4phk2 |
AddedInterface |
Add eth0 [10.132.0.51/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fb75d-predictor-768dc4cffb-4phk2 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fb75d-predictor-768dc4cffb-4phk2 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-fb75d-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-fb75d-predictor-768dc4cffb from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-fb75d-predictor-768dc4cffb |
SuccessfulCreate |
Created pod: raw-sklearn-fb75d-predictor-768dc4cffb-4phk2 | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fb75d-predictor-768dc4cffb-4phk2 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-fb75d |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-fb75d": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-fb75d": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fb75d-predictor-768dc4cffb-4phk2 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fb75d-predictor-768dc4cffb-4phk2 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fb75d-predictor-768dc4cffb-4phk2 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fb75d-predictor-768dc4cffb-4phk2 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fb75d-predictor-768dc4cffb-4phk2 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fb75d-predictor-768dc4cffb-4phk2 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-fb75d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-fb75d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-fb75d |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-fb75d |
InferenceServiceReady |
InferenceService [raw-sklearn-fb75d] is Ready | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-fb75d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-fb75d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fb75d-predictor-768dc4cffb-4phk2 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fb75d-predictor-768dc4cffb-4phk2 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-fb75d-predictor-768dc4cffb-4phk2 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.51:8643/healthz": dial tcp 10.132.0.51:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-fb75d-predictor-768dc4cffb-4phk2 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.51:8080: connect: connection refused |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-runtime-79274-predictor-58c54cc568 |
SuccessfulCreate |
Created pod: raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-79274 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-79274": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx |
AddedInterface |
Add eth0 [10.132.0.52/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-runtime-79274-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-runtime-79274-predictor-58c54cc568 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-79274 |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-runtime-79274": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-79274": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx |
Created |
Created container: kserve-container | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-79274-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-79274-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-79274 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-79274 |
InferenceServiceReady |
InferenceService [raw-sklearn-runtime-79274] is Ready | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx |
Killing |
Stopping container kube-rbac-proxy | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-79274-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-79274-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx |
Killing |
Stopping container kserve-container | |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.52:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-79274-predictor-58c54cc568-s79tx |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.52:8643/healthz": dial tcp 10.132.0.52:8643: connect: connection refused |