| Time | Namespace | Component | RelatedObject | Reason | Message |
|---|---|---|---|---|---|
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m to ip-10-0-132-110.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz to ip-10-0-132-110.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-1b651-predictor-55d656b769-2v9mr |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr to ip-10-0-132-110.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9 to ip-10-0-135-126.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-74e2d-864c794844-n9cgw |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw to ip-10-0-135-126.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp to ip-10-0-132-110.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt to ip-10-0-132-110.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-predictor-69fc457c9b-qg8bw |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw to ip-10-0-132-110.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-v2-predictor-5957db6ddf-2bz7c |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c to ip-10-0-132-110.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp to ip-10-0-132-110.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-bb95f9bbd-wglnd |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd to ip-10-0-132-110.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk to ip-10-0-135-126.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-predictor-cf655cf76-925g5 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5 to ip-10-0-132-110.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd to ip-10-0-132-110.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r to ip-10-0-132-110.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455 to ip-10-0-132-110.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq to ip-10-0-132-110.ec2.internal | ||
kserve-ci-e2e-test |
isvc-init-fail-505b2f-predictor-5df49d5965-gptlx |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx to ip-10-0-132-110.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht to ip-10-0-132-110.ec2.internal | ||
kserve-ci-e2e-test |
isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs to ip-10-0-132-110.ec2.internal | ||
kserve-ci-e2e-test |
isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j to ip-10-0-132-110.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p to ip-10-0-135-126.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-bb95f9bbd-dzj7q |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q to ip-10-0-135-126.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-predictor-698bbb5b56-nctjt |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt to ip-10-0-132-110.ec2.internal | ||
kserve-ci-e2e-test |
isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t to ip-10-0-132-110.ec2.internal | ||
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-9d91d |
UpdateFailed |
Failed to update status for InferenceService "isvc-raw-sklearn-batcher-9d91d": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-9d91d": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j |
AddedInterface |
Add eth0 [10.134.0.26/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c |
SuccessfulCreate |
Created pod: isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j | |
kserve-ci-e2e-test |
deployment-controller |
isvc-raw-sklearn-batcher-9d91d-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-9d91d |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-9d91d": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1813" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1813" in 13.064s (13.064s including waiting). Image size: 1555733417 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.276s (2.276s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j |
Pulling |
Pulling image "quay.io/opendatahub/kserve-agent@sha256:7789d265344a8abbbdf46d39305389c0ec3159d12d2d92e2c5830f8aeb79d1f1" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:7789d265344a8abbbdf46d39305389c0ec3159d12d2d92e2c5830f8aeb79d1f1" in 2.363s (2.363s including waiting). Image size: 240170086 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j |
Started |
Started container agent | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-9d91d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-9d91d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-9d91d |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-9d91d |
InferenceServiceReady |
InferenceService [isvc-raw-sklearn-batcher-9d91d] is Ready | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-9d91d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-74e2d-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-74e2d |
UpdateFailed |
Failed to update status for InferenceService "isvc-xgboost-graph-raw-74e2d": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-graph-raw-74e2d": the object has been modified; please apply your changes to the latest version and try again | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-9d91d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-graph-raw-74e2d-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-graph-raw-74e2d-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-74e2d |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-74e2d": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-74e2d": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-74e2d-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp |
AddedInterface |
Add eth0 [10.134.0.28/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m |
AddedInterface |
Add eth0 [10.134.0.27/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1813" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m |
Started |
Started container kserve-container | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.26:8080: connect: connection refused |
| (x3) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.26:8643/healthz": dial tcp 10.134.0.26:8643: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 17.222s (17.222s including waiting). Image size: 1307355896 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp |
Created |
Created container: kube-rbac-proxy | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-74e2d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-74e2d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-74e2d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-74e2d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x7) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.28:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-74e2d |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-74e2d |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-74e2d] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-74e2d |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-74e2d |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-74e2d] is Ready | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-74e2d-864c794844 |
SuccessfulCreate |
Created pod: model-chainer-raw-74e2d-864c794844-n9cgw | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-74e2d |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-74e2d] is Ready | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-74e2d-864c794844-n9cgw |
AddedInterface |
Add eth0 [10.133.0.34/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-74e2d-864c794844-n9cgw |
Pulling |
Pulling image "quay.io/opendatahub/kserve-router@sha256:bcb85ef6ed85b7da131182bb5f673741b67400a2637d62d1dcee3f3c438674f0" | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-74e2d |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-74e2d-864c794844 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-74e2d-864c794844-n9cgw |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:bcb85ef6ed85b7da131182bb5f673741b67400a2637d62d1dcee3f3c438674f0" in 2.098s (2.098s including waiting). Image size: 218226791 bytes. | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-74e2d-864c794844-n9cgw |
Created |
Created container: model-chainer-raw-74e2d | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-74e2d-864c794844-n9cgw |
Started |
Started container model-chainer-raw-74e2d | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-74e2d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-74e2d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-74e2d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-74e2d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-graph-raw-hpa-b0501-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-graph-raw-hpa-b0501-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-74e2d-864c794844-n9cgw |
Killing |
Stopping container model-chainer-raw-74e2d | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-hpa-b0501-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f from 0 to 1 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-hpa-b0501-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt |
AddedInterface |
Add eth0 [10.134.0.30/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq |
AddedInterface |
Add eth0 [10.134.0.29/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.27:8643/healthz": dial tcp 10.134.0.27:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.27:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt |
Pulled |
Container image "kserve/xgbserver:latest" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1813" already present on machine | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-74e2d-864c794844-n9cgw |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-b0501-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-b0501-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-b0501-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-b0501-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-b0501 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-hpa-b0501] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-b0501 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-b0501 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-hpa-b0501] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-b0501 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-raw-hpa-b0501-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-hpa-b0501-578b7cf5fb |
SuccessfulCreate |
Created pod: model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9 | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-hpa-b0501 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-hpa-b0501-578b7cf5fb from 0 to 1 | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-b0501 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-hpa-b0501] is Ready | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-b0501 |
UpdateFailed |
Failed to update status for InferenceGraph "model-chainer-raw-hpa-b0501": Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "model-chainer-raw-hpa-b0501": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-b0501 |
InternalError |
fails to update InferenceGraph status: Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "model-chainer-raw-hpa-b0501": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9 |
Started |
Started container model-chainer-raw-hpa-b0501 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9 |
Created |
Created container: model-chainer-raw-hpa-b0501 | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9 |
AddedInterface |
Add eth0 [10.133.0.35/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9 |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:bcb85ef6ed85b7da131182bb5f673741b67400a2637d62d1dcee3f3c438674f0" already present on machine | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-b0501-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-b0501-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-b0501-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-b0501-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9 |
Killing |
Stopping container model-chainer-raw-hpa-b0501 | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-8e237 |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-8e237": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-8e237": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-8e237-predictor-8c874fbcc |
SuccessfulCreate |
Created pod: message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-8e237-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-8e237-predictor-8c874fbcc from 0 to 1 | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.29:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "message-dumper-raw-8e237-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp |
AddedInterface |
Add eth0 [10.134.0.31/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp |
Pulling |
Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.29:8643/healthz": dial tcp 10.134.0.29:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp |
Pulled |
Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.165s (1.165s including waiting). Image size: 14813193 bytes. | |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.30:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.30:8643/healthz": dial tcp 10.134.0.30:8643: connect: connection refused | |
| (x6) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-8e237 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-8e237 |
InferenceServiceReady |
InferenceService [message-dumper-raw-8e237] is Ready | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-8e237 |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-8e237": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-8e237": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-8e237-predictor-557d6ccdb9 |
SuccessfulCreate |
Created pod: isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht | |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-8e237-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-8e237-predictor-557d6ccdb9 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht |
AddedInterface |
Add eth0 [10.134.0.32/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1813" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:7789d265344a8abbbdf46d39305389c0ec3159d12d2d92e2c5830f8aeb79d1f1" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht |
Created |
Created container: agent | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-8e237-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-8e237-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-8e237-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-8e237-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-8e237 |
InferenceServiceReady |
InferenceService [isvc-logger-raw-8e237] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-8e237 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-8e237-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-8e237-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-scale-raw-d2f58-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-scale-raw-d2f58-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-d2f58 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-scale-raw-d2f58": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scale-raw-d2f58": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5 |
SuccessfulCreate |
Created pod: isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455 |
AddedInterface |
Add eth0 [10.134.0.33/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1813" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455 |
Started |
Started container kserve-container | |
| (x4) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.32:8643/healthz": dial tcp 10.134.0.32:8643: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.32:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-d2f58-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-d2f58-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-d2f58 |
InferenceServiceReady |
InferenceService [isvc-sklearn-scale-raw-d2f58] is Ready | |
| (x9) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-d2f58-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-d2f58 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x9) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-d2f58-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455 |
Killing |
Stopping container kube-rbac-proxy | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-create": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5957db6ddf-2bz7c |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-create-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-create-predictor-69fc457c9b from 0 to 1 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-create-v2-predictor-5957db6ddf from 0 to 1 | |
| (x20) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.33:8080: connect: connection refused |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-v2-predictor-5957db6ddf |
SuccessfulCreate |
Created pod: isvc-canary-create-v2-predictor-5957db6ddf-2bz7c | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-predictor-69fc457c9b |
SuccessfulCreate |
Created pod: isvc-canary-create-predictor-69fc457c9b-qg8bw | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-69fc457c9b-qg8bw |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5957db6ddf-2bz7c |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-v2-predictor-5957db6ddf-2bz7c |
AddedInterface |
Add eth0 [10.134.0.35/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5957db6ddf-2bz7c |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5957db6ddf-2bz7c |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-69fc457c9b-qg8bw |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-69fc457c9b-qg8bw |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-predictor-69fc457c9b-qg8bw |
AddedInterface |
Add eth0 [10.134.0.34/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.33:8643/healthz": dial tcp 10.134.0.33:8643: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-69fc457c9b-qg8bw |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1813" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5957db6ddf-2bz7c |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1813" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5957db6ddf-2bz7c |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-69fc457c9b-qg8bw |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5957db6ddf-2bz7c |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5957db6ddf-2bz7c |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5957db6ddf-2bz7c |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5957db6ddf-2bz7c |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-69fc457c9b-qg8bw |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-69fc457c9b-qg8bw |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-69fc457c9b-qg8bw |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-69fc457c9b-qg8bw |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
InferenceServiceReady |
InferenceService [isvc-canary-create] is Ready | |
| (x15) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-69fc457c9b-qg8bw |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-69fc457c9b-qg8bw |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5957db6ddf-2bz7c |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5957db6ddf-2bz7c |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-69fc457c9b-qg8bw |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.34:8643/healthz": dial tcp 10.134.0.34:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-69fc457c9b-qg8bw |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.34:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5957db6ddf-2bz7c |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.35:8643/healthz": dial tcp 10.134.0.35:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-5957db6ddf-2bz7c |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.35:8080: connect: connection refused |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-predictor-698bbb5b56 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-predictor-698bbb5b56 |
SuccessfulCreate |
Created pod: isvc-canary-promote-predictor-698bbb5b56-nctjt | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-v2-predictor-74bdc8ffff from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-v2-predictor-74bdc8ffff |
SuccessfulCreate |
Created pod: isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r |
AddedInterface |
Add eth0 [10.134.0.37/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-698bbb5b56-nctjt |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-698bbb5b56-nctjt |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-698bbb5b56-nctjt |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-predictor-698bbb5b56-nctjt |
AddedInterface |
Add eth0 [10.134.0.36/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-698bbb5b56-nctjt |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-promote-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1813" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-698bbb5b56-nctjt |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-698bbb5b56-nctjt |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-698bbb5b56-nctjt |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-698bbb5b56-nctjt |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-698bbb5b56-nctjt |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-698bbb5b56-nctjt |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1813" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-698bbb5b56-nctjt |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-698bbb5b56-nctjt |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-v2-predictor-74bdc8ffff from 1 to 2 | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-promote": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again |
| (x20) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf" | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-v2-predictor-74bdc8ffff |
SuccessfulCreate |
Created pod: isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p |
AddedInterface |
Add eth0 [10.133.0.36/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InferenceServiceNotReady |
InferenceService [isvc-canary-promote] is no longer Ready because of: PredictorReady, Ready, Stopped | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-698bbb5b56-nctjt |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.36:8643/healthz": dial tcp 10.134.0.36:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-698bbb5b56-nctjt |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.36:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf" in 3.451s (3.451s including waiting). Image size: 302621120 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1813" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1813" in 13.389s (13.389s including waiting). Image size: 1555733417 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.227s (2.227s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p |
Created |
Created container: kube-rbac-proxy | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InferenceServiceReady |
InferenceService [isvc-canary-promote] is Ready |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p |
Killing |
Stopping container kube-rbac-proxy | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.36:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.37:8643/healthz": dial tcp 10.134.0.37:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.36:8643/healthz": dial tcp 10.133.0.36:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.37:8080: connect: connection refused |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-v2-predictor-7799dfb7df from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-bb95f9bbd |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-bb95f9bbd-wglnd | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-predictor-bb95f9bbd from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-v2-predictor-7799dfb7df |
SuccessfulCreate |
Created pod: isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk |
AddedInterface |
Add eth0 [10.133.0.37/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-wglnd |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-predictor-serving-cert" not found |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-wglnd |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-wglnd |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-wglnd |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-bb95f9bbd-wglnd |
AddedInterface |
Add eth0 [10.134.0.38/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1813" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-wglnd |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-wglnd |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-wglnd |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-wglnd |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-wglnd |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-wglnd |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1813" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-dzj7q |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-dzj7q |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-dzj7q |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-bb95f9bbd-dzj7q |
AddedInterface |
Add eth0 [10.133.0.38/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-bb95f9bbd |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-bb95f9bbd-dzj7q | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-predictor-bb95f9bbd from 1 to 2 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InferenceServiceNotReady |
InferenceService [isvc-canary-rollback] is no longer Ready because of: PredictorReady, Ready, Stopped | |
| (x20) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-rollback": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.37:8643/healthz": dial tcp 10.133.0.37:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.37:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-dzj7q |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-dzj7q |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1813" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-dzj7q |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-dzj7q |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-dzj7q |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-dzj7q |
Started |
Started container kube-rbac-proxy | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InferenceServiceReady |
InferenceService [isvc-canary-rollback] is Ready |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-dzj7q |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-wglnd |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-wglnd |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-dzj7q |
Killing |
Stopping container kube-rbac-proxy | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-dzj7q |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.38:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-dzj7q |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.38:8643/healthz": dial tcp 10.133.0.38:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-wglnd |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.38:8643/healthz": dial tcp 10.134.0.38:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-bb95f9bbd-wglnd |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.38:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-stop": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-stop-predictor-cf655cf76 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-v2-predictor-75fcd5578f |
SuccessfulCreate |
Created pod: isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-predictor-cf655cf76 |
SuccessfulCreate |
Created pod: isvc-canary-stop-predictor-cf655cf76-925g5 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-stop-v2-predictor-75fcd5578f from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-cf655cf76-925g5 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-cf655cf76-925g5 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-cf655cf76-925g5 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd |
AddedInterface |
Add eth0 [10.134.0.40/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-predictor-cf655cf76-925g5 |
AddedInterface |
Add eth0 [10.134.0.39/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-cf655cf76-925g5 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-cf655cf76-925g5 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1813" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-cf655cf76-925g5 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-cf655cf76-925g5 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-cf655cf76-925g5 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-cf655cf76-925g5 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1813" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InferenceServiceReady |
InferenceService [isvc-canary-stop] is Ready | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InferenceServiceNotReady |
InferenceService [isvc-canary-stop] is no longer Ready because of: PredictorReady, Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd |
Killing |
Stopping container kube-rbac-proxy | |
| (x19) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-cf655cf76-925g5 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-cf655cf76-925g5 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.40:8643/healthz": dial tcp 10.134.0.40:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-cf655cf76-925g5 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.39:8643/healthz": dial tcp 10.134.0.39:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-cf655cf76-925g5 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.39:8080: connect: connection refused |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.40:8080: connect: connection refused |
kserve-ci-e2e-test |
deployment-controller |
isvc-primary-1e74d8-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-primary-1e74d8-predictor-6dfc9b49f6 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-1e74d8 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-1e74d8": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-primary-1e74d8-predictor-6dfc9b49f6 |
SuccessfulCreate |
Created pod: isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs |
AddedInterface |
Add eth0 [10.134.0.41/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-1e74d8 |
UpdateFailed |
Failed to update status for InferenceService "isvc-primary-1e74d8": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-1e74d8": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1813" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs |
Started |
Started container kserve-container | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-1e74d8-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-1e74d8-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.41:8080: connect: connection refused |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-1e74d8 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-1e74d8 |
InferenceServiceReady |
InferenceService [isvc-primary-1e74d8] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-secondary-1e74d8-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-1e74d8 |
UpdateFailed |
Failed to update status for InferenceService "isvc-secondary-1e74d8": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-1e74d8": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-secondary-1e74d8-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-secondary-1e74d8-predictor-7f998fd97f from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-1e74d8 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-1e74d8": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-secondary-1e74d8-predictor-7f998fd97f |
SuccessfulCreate |
Created pod: isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t | |
kserve-ci-e2e-test |
multus |
isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t |
AddedInterface |
Add eth0 [10.134.0.42/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf" already present on machine |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t |
Created |
Created container: storage-initializer |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-1e74d8 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t_kserve-ci-e2e-test(33eb3664-13f0-4c97-933f-d263733467f1) | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-1e74d8-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-1e74d8-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-1e74d8-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-1e74d8-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-505b2f |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-505b2f": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-505b2f |
UpdateFailed |
Failed to update status for InferenceService "isvc-init-fail-505b2f": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-505b2f": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-init-fail-505b2f-predictor-5df49d5965-gptlx |
AddedInterface |
Add eth0 [10.134.0.43/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-init-fail-505b2f-predictor-5df49d5965 |
SuccessfulCreate |
Created pod: isvc-init-fail-505b2f-predictor-5df49d5965-gptlx | |
kserve-ci-e2e-test |
deployment-controller |
isvc-init-fail-505b2f-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-init-fail-505b2f-predictor-5df49d5965 from 0 to 1 | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-505b2f-predictor-5df49d5965-gptlx |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-505b2f-predictor-5df49d5965-gptlx |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-505b2f-predictor-5df49d5965-gptlx |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf" already present on machine |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-505b2f |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-505b2f-predictor-5df49d5965-gptlx |
Killing |
Stopping container storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-1b651-predictor-55d656b769 |
SuccessfulCreate |
Created pod: raw-sklearn-1b651-predictor-55d656b769-2v9mr | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-1b651 |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-1b651": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-1b651": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-1b651-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-1b651-predictor-55d656b769 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-1b651-predictor-55d656b769-2v9mr |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf" already present on machine | |
kserve-ci-e2e-test |
multus |
raw-sklearn-1b651-predictor-55d656b769-2v9mr |
AddedInterface |
Add eth0 [10.134.0.44/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-1b651-predictor-55d656b769-2v9mr |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-1b651-predictor-55d656b769-2v9mr |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-1b651-predictor-55d656b769-2v9mr |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-1b651-predictor-55d656b769-2v9mr |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-1b651-predictor-55d656b769-2v9mr |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-1b651-predictor-55d656b769-2v9mr |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-1b651-predictor-55d656b769-2v9mr |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-1b651-predictor-55d656b769-2v9mr |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1813" already present on machine | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-1b651-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-1b651-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-1b651 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-1b651 |
InferenceServiceReady |
InferenceService [raw-sklearn-1b651] is Ready | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-1b651-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-1b651-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-1b651-predictor-55d656b769-2v9mr |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-1b651-predictor-55d656b769-2v9mr |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-1b651-predictor-55d656b769-2v9mr |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.44:8643/healthz": dial tcp 10.134.0.44:8643: connect: connection refused | |
| (x9) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-1b651-predictor-55d656b769-2v9mr |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.44:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-17dc9 |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-runtime-17dc9": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-17dc9": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-runtime-17dc9-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-runtime-17dc9-predictor-6984f87bcc from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-runtime-17dc9-predictor-6984f87bcc |
SuccessfulCreate |
Created pod: raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "raw-sklearn-runtime-17dc9-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
multus |
raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz |
AddedInterface |
Add eth0 [10.134.0.45/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:527f9c4ef06f4e1d255c6c4b87333be01c67680fe0b3417b8fb7aeb5d5b61bbf" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1813" already present on machine | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-17dc9-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-17dc9-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-17dc9 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-17dc9 |
InferenceServiceReady |
InferenceService [raw-sklearn-runtime-17dc9] is Ready | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz |
Killing |
Stopping container kserve-container | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-17dc9-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-17dc9-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.45:8643/healthz": dial tcp 10.134.0.45:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.45:8080: connect: connection refused |