| Time | Namespace | Component | RelatedObject | Reason | Message |
|---|---|---|---|---|---|
kserve-ci-e2e-test |
isvc-sklearn-scale-raw-e81d6-predictor-6c8678b65-zmk7d |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-scale-raw-e81d6-predictor-6c8678b65-zmk7d to ip-10-0-132-64.ec2.internal | ||
kserve-ci-e2e-test |
isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 to ip-10-0-132-64.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-926cf-predictor-6fc8899c49-49qgj |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-926cf-predictor-6fc8899c49-49qgj to ip-10-0-132-64.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-hpa-3f0c4-868c7dd7d5-g9vg8 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-hpa-3f0c4-868c7dd7d5-g9vg8 to ip-10-0-132-64.ec2.internal | ||
kserve-ci-e2e-test |
isvc-primary-bbf281-predictor-6b7cf7ddb4-9mhhw |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-primary-bbf281-predictor-6b7cf7ddb4-9mhhw to ip-10-0-132-64.ec2.internal | ||
kserve-ci-e2e-test |
isvc-init-fail-8448e3-predictor-84676484d9-mvhgn |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-init-fail-8448e3-predictor-84676484d9-mvhgn to ip-10-0-132-64.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-runtime-bca52-predictor-7cc55b85d5-nl97d |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-runtime-bca52-predictor-7cc55b85d5-nl97d to ip-10-0-132-64.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-3b499-predictor-756fc45b6c-ffgpv |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b499-predictor-756fc45b6c-ffgpv to ip-10-0-132-64.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-hpa-3f0c4-predictor-76f85bb77c-2s7tb |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-3f0c4-predictor-76f85bb77c-2s7tb to ip-10-0-139-62.ec2.internal | ||
kserve-ci-e2e-test |
isvc-secondary-bbf281-predictor-7f76d5b559-s84pm |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-secondary-bbf281-predictor-7f76d5b559-s84pm to ip-10-0-132-64.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-3b499-predictor-f94984f49-pzpv7 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b499-predictor-f94984f49-pzpv7 to ip-10-0-139-62.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-b8788-predictor-6cfdc7f55c-95t5k |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-b8788-predictor-6cfdc7f55c-95t5k to ip-10-0-132-64.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-hpa-3f0c4-predictor-759c5664f9-trsgt |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-3f0c4-predictor-759c5664f9-trsgt to ip-10-0-132-64.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-3b499-57f97cb5b9-zc5fk |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-3b499-57f97cb5b9-zc5fk to ip-10-0-132-64.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-b8788-predictor-85565b99f9-hckmm |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-b8788-predictor-85565b99f9-hckmm to ip-10-0-139-62.ec2.internal | ||
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-8fb8f |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-8fb8f": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9 |
SuccessfulCreate |
Created pod: isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 | |
kserve-ci-e2e-test |
multus |
isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 |
AddedInterface |
Add eth0 [10.134.0.37/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-8fb8f |
UpdateFailed |
Failed to update status for InferenceService "isvc-raw-sklearn-batcher-8fb8f": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-8fb8f": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-raw-sklearn-batcher-8fb8f-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:72e9a53b4f6609f050c864ccf8bba3bc53b15bbbb0a09e5de74a3bdc7465dd84" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:72e9a53b4f6609f050c864ccf8bba3bc53b15bbbb0a09e5de74a3bdc7465dd84" in 2.881s (2.881s including waiting). Image size: 301492132 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1435" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1435" in 13.759s (13.759s including waiting). Image size: 1560924583 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.398s (2.398s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-agent@sha256:e376846d2aa0b7fb92a16b8c1edcc05bf3c05895534eec2581922448b745f27a" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:e376846d2aa0b7fb92a16b8c1edcc05bf3c05895534eec2581922448b745f27a" in 2.576s (2.576s including waiting). Image size: 238035014 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 |
Started |
Started container agent | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-8fb8f-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-8fb8f-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-8fb8f |
InferenceServiceReady |
InferenceService [isvc-raw-sklearn-batcher-8fb8f] is Ready | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-8fb8f-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-8fb8f-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-8fb8f |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-3b499-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-3b499-predictor-f94984f49 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-3b499 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-3b499": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-3b499": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-3b499-predictor-f94984f49 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-3b499-predictor-f94984f49-pzpv7 | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-3b499-predictor-756fc45b6c |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-3b499-predictor-756fc45b6c-ffgpv | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-3b499-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-3b499-predictor-756fc45b6c from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-3b499-predictor-756fc45b6c-ffgpv |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-3b499-predictor-f94984f49-pzpv7 |
AddedInterface |
Add eth0 [10.132.0.25/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-3b499-predictor-f94984f49-pzpv7 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:72e9a53b4f6609f050c864ccf8bba3bc53b15bbbb0a09e5de74a3bdc7465dd84" | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-3b499-predictor-756fc45b6c-ffgpv |
AddedInterface |
Add eth0 [10.134.0.38/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-3b499-predictor-756fc45b6c-ffgpv |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:72e9a53b4f6609f050c864ccf8bba3bc53b15bbbb0a09e5de74a3bdc7465dd84" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-3b499-predictor-756fc45b6c-ffgpv |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-3b499-predictor-f94984f49-pzpv7 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-3b499-predictor-f94984f49-pzpv7 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-3b499-predictor-f94984f49-pzpv7 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:72e9a53b4f6609f050c864ccf8bba3bc53b15bbbb0a09e5de74a3bdc7465dd84" in 3.098s (3.098s including waiting). Image size: 301492132 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-3b499-predictor-756fc45b6c-ffgpv |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1435" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-3b499-predictor-756fc45b6c-ffgpv |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-3b499-predictor-756fc45b6c-ffgpv |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-3b499-predictor-756fc45b6c-ffgpv |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-3b499-predictor-756fc45b6c-ffgpv |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-3b499-predictor-756fc45b6c-ffgpv |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-3b499-predictor-f94984f49-pzpv7 |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-3b499-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.37:8080: connect: connection refused |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-3b499-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x4) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-8fb8f-predictor-59f6ff8c9-44d99 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.37:8643/healthz": dial tcp 10.134.0.37:8643: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-3b499-predictor-f94984f49-pzpv7 |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 19.251s (19.251s including waiting). Image size: 1306417402 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-3b499-predictor-f94984f49-pzpv7 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-3b499-predictor-f94984f49-pzpv7 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-3b499-predictor-f94984f49-pzpv7 |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-3b499-predictor-f94984f49-pzpv7 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-3b499-predictor-f94984f49-pzpv7 |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.292s (2.292s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-3b499-predictor-f94984f49-pzpv7 |
Created |
Created container: kube-rbac-proxy | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-3b499-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-3b499-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-3b499 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-3b499] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-3b499 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-3b499-predictor-f94984f49-pzpv7 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.25:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-3b499 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-3b499 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-3b499] is Ready | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-3b499 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-3b499] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-3b499 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-3b499-57f97cb5b9 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-3b499-57f97cb5b9 |
SuccessfulCreate |
Created pod: model-chainer-raw-3b499-57f97cb5b9-zc5fk | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-3b499-57f97cb5b9-zc5fk |
Pulling |
Pulling image "quay.io/opendatahub/kserve-router@sha256:8608123ddb1fc42bff2e1c9ce408ce175b3696948e68bd4ec98a77fff89c49f4" | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-3b499-57f97cb5b9-zc5fk |
AddedInterface |
Add eth0 [10.134.0.39/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-3b499-57f97cb5b9-zc5fk |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:8608123ddb1fc42bff2e1c9ce408ce175b3696948e68bd4ec98a77fff89c49f4" in 2.335s (2.336s including waiting). Image size: 216346185 bytes. | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-3b499-57f97cb5b9-zc5fk |
Created |
Created container: model-chainer-raw-3b499 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-3b499-57f97cb5b9-zc5fk |
Started |
Started container model-chainer-raw-3b499 | |
| (x6) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-3b499-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x6) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-3b499-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-3b499-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-3b499-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-3f0c4 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-hpa-3f0c4": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-hpa-3f0c4": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-hpa-3f0c4-predictor-759c5664f9 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-hpa-3f0c4-predictor-759c5664f9-trsgt | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-3b499-predictor-756fc45b6c-ffgpv |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-3b499-57f97cb5b9-zc5fk |
Killing |
Stopping container model-chainer-raw-3b499 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-3b499-predictor-756fc45b6c-ffgpv |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-hpa-3f0c4-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-hpa-3f0c4-predictor-759c5664f9 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-3b499-predictor-f94984f49-pzpv7 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3f0c4-predictor-759c5664f9-trsgt |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-graph-raw-hpa-3f0c4-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-hpa-3f0c4-predictor-76f85bb77c-2s7tb |
AddedInterface |
Add eth0 [10.132.0.26/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-hpa-3f0c4-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-hpa-3f0c4-predictor-76f85bb77c from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-hpa-3f0c4-predictor-76f85bb77c |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-hpa-3f0c4-predictor-76f85bb77c-2s7tb | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-3f0c4 |
UpdateFailed |
Failed to update status for InferenceService "isvc-xgboost-graph-raw-hpa-3f0c4": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-graph-raw-hpa-3f0c4": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3f0c4-predictor-76f85bb77c-2s7tb |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:72e9a53b4f6609f050c864ccf8bba3bc53b15bbbb0a09e5de74a3bdc7465dd84" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3f0c4-predictor-76f85bb77c-2s7tb |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3f0c4-predictor-76f85bb77c-2s7tb |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-3b499-predictor-f94984f49-pzpv7 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-3b499-predictor-f94984f49-pzpv7 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.25:8643/healthz": dial tcp 10.132.0.25:8643: connect: connection refused | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-hpa-3f0c4-predictor-759c5664f9-trsgt |
AddedInterface |
Add eth0 [10.134.0.40/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3f0c4-predictor-759c5664f9-trsgt |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3f0c4-predictor-759c5664f9-trsgt |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:72e9a53b4f6609f050c864ccf8bba3bc53b15bbbb0a09e5de74a3bdc7465dd84" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3f0c4-predictor-759c5664f9-trsgt |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-3b499-predictor-756fc45b6c-ffgpv |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.38:8643/healthz": dial tcp 10.134.0.38:8643: connect: connection refused | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-3b499-predictor-756fc45b6c-ffgpv |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.38:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3f0c4-predictor-76f85bb77c-2s7tb |
Pulled |
Container image "kserve/xgbserver:latest" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3f0c4-predictor-759c5664f9-trsgt |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3f0c4-predictor-76f85bb77c-2s7tb |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3f0c4-predictor-759c5664f9-trsgt |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1435" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3f0c4-predictor-759c5664f9-trsgt |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3f0c4-predictor-759c5664f9-trsgt |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3f0c4-predictor-76f85bb77c-2s7tb |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3f0c4-predictor-76f85bb77c-2s7tb |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3f0c4-predictor-76f85bb77c-2s7tb |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3f0c4-predictor-76f85bb77c-2s7tb |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3f0c4-predictor-759c5664f9-trsgt |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3f0c4-predictor-759c5664f9-trsgt |
Started |
Started container kube-rbac-proxy | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-3f0c4-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-3f0c4-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-3b499-57f97cb5b9-zc5fk |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-3f0c4-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-3f0c4-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x8) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-3f0c4 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-3f0c4 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-hpa-3f0c4] is Ready | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-3f0c4 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-hpa-3f0c4] is Ready | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-3f0c4 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-3f0c4-868c7dd7d5-g9vg8 |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:8608123ddb1fc42bff2e1c9ce408ce175b3696948e68bd4ec98a77fff89c49f4" already present on machine | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-hpa-3f0c4-868c7dd7d5-g9vg8 |
AddedInterface |
Add eth0 [10.134.0.41/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-3f0c4 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-hpa-3f0c4] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-hpa-3f0c4 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-hpa-3f0c4-868c7dd7d5 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-hpa-3f0c4-868c7dd7d5 |
SuccessfulCreate |
Created pod: model-chainer-raw-hpa-3f0c4-868c7dd7d5-g9vg8 | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-3f0c4-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-3f0c4-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-3f0c4-868c7dd7d5-g9vg8 |
Started |
Started container model-chainer-raw-hpa-3f0c4 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-3f0c4-868c7dd7d5-g9vg8 |
Created |
Created container: model-chainer-raw-hpa-3f0c4 | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-3f0c4-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-3f0c4-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-b8788-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-b8788-predictor-85565b99f9 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-3f0c4-868c7dd7d5-g9vg8 |
Killing |
Stopping container model-chainer-raw-hpa-3f0c4 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3f0c4-predictor-759c5664f9-trsgt |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3f0c4-predictor-76f85bb77c-2s7tb |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3f0c4-predictor-76f85bb77c-2s7tb |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-b8788-predictor-85565b99f9 |
SuccessfulCreate |
Created pod: message-dumper-raw-b8788-predictor-85565b99f9-hckmm | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3f0c4-predictor-759c5664f9-trsgt |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-b8788-predictor-85565b99f9-hckmm |
Pulling |
Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-b8788 |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-b8788": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-b8788": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-b8788-predictor-85565b99f9-hckmm |
AddedInterface |
Add eth0 [10.132.0.27/23] from ovn-kubernetes | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3f0c4-predictor-76f85bb77c-2s7tb |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.26:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-3f0c4-predictor-76f85bb77c-2s7tb |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.26:8643/healthz": dial tcp 10.132.0.26:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-b8788-predictor-85565b99f9-hckmm |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-b8788-predictor-85565b99f9-hckmm |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-b8788-predictor-85565b99f9-hckmm |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-b8788-predictor-85565b99f9-hckmm |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-b8788-predictor-85565b99f9-hckmm |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-b8788-predictor-85565b99f9-hckmm |
Pulled |
Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.105s (1.105s including waiting). Image size: 14813193 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3f0c4-predictor-759c5664f9-trsgt |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.40:8643/healthz": dial tcp 10.134.0.40:8643: connect: connection refused | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-3f0c4-predictor-759c5664f9-trsgt |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.40:8080: connect: connection refused |
| (x7) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-b8788 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-b8788 |
InferenceServiceReady |
InferenceService [message-dumper-raw-b8788] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8788-predictor-6cfdc7f55c-95t5k |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-logger-raw-b8788-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-b8788-predictor-6cfdc7f55c |
SuccessfulCreate |
Created pod: isvc-logger-raw-b8788-predictor-6cfdc7f55c-95t5k | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-b8788 |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-b8788": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-b8788": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-b8788-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-b8788-predictor-6cfdc7f55c from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8788-predictor-6cfdc7f55c-95t5k |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8788-predictor-6cfdc7f55c-95t5k |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:72e9a53b4f6609f050c864ccf8bba3bc53b15bbbb0a09e5de74a3bdc7465dd84" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-b8788-predictor-6cfdc7f55c-95t5k |
AddedInterface |
Add eth0 [10.134.0.42/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8788-predictor-6cfdc7f55c-95t5k |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8788-predictor-6cfdc7f55c-95t5k |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1435" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8788-predictor-6cfdc7f55c-95t5k |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8788-predictor-6cfdc7f55c-95t5k |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:e376846d2aa0b7fb92a16b8c1edcc05bf3c05895534eec2581922448b745f27a" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8788-predictor-6cfdc7f55c-95t5k |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8788-predictor-6cfdc7f55c-95t5k |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8788-predictor-6cfdc7f55c-95t5k |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8788-predictor-6cfdc7f55c-95t5k |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8788-predictor-6cfdc7f55c-95t5k |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8788-predictor-6cfdc7f55c-95t5k |
Created |
Created container: agent | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-3f0c4-868c7dd7d5-g9vg8 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-b8788-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-b8788-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-b8788-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-b8788-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-b8788 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-b8788 |
InferenceServiceReady |
InferenceService [isvc-logger-raw-b8788] is Ready | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-b8788-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-b8788-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-e81d6 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-scale-raw-e81d6": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scale-raw-e81d6": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-scale-raw-e81d6-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-scale-raw-e81d6-predictor-6c8678b65 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8788-predictor-6cfdc7f55c-95t5k |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8788-predictor-6cfdc7f55c-95t5k |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-scale-raw-e81d6-predictor-6c8678b65 |
SuccessfulCreate |
Created pod: isvc-sklearn-scale-raw-e81d6-predictor-6c8678b65-zmk7d | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e81d6-predictor-6c8678b65-zmk7d |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-scale-raw-e81d6-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-b8788-predictor-85565b99f9-hckmm |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8788-predictor-6cfdc7f55c-95t5k |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-b8788-predictor-85565b99f9-hckmm |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e81d6-predictor-6c8678b65-zmk7d |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:72e9a53b4f6609f050c864ccf8bba3bc53b15bbbb0a09e5de74a3bdc7465dd84" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-scale-raw-e81d6-predictor-6c8678b65-zmk7d |
AddedInterface |
Add eth0 [10.134.0.43/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e81d6-predictor-6c8678b65-zmk7d |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e81d6-predictor-6c8678b65-zmk7d |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e81d6-predictor-6c8678b65-zmk7d |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e81d6-predictor-6c8678b65-zmk7d |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e81d6-predictor-6c8678b65-zmk7d |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e81d6-predictor-6c8678b65-zmk7d |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e81d6-predictor-6c8678b65-zmk7d |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e81d6-predictor-6c8678b65-zmk7d |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1435" already present on machine | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8788-predictor-6cfdc7f55c-95t5k |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x4) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8788-predictor-6cfdc7f55c-95t5k |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.42:8643/healthz": dial tcp 10.134.0.42:8643: connect: connection refused |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-b8788-predictor-6cfdc7f55c-95t5k |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.42:8080: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-e81d6-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-e81d6-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-e81d6 |
InferenceServiceReady |
InferenceService [isvc-sklearn-scale-raw-e81d6] is Ready | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-e81d6 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x8) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-e81d6-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
replicaset-controller |
isvc-primary-bbf281-predictor-6b7cf7ddb4 |
SuccessfulCreate |
Created pod: isvc-primary-bbf281-predictor-6b7cf7ddb4-9mhhw | |
kserve-ci-e2e-test |
deployment-controller |
isvc-primary-bbf281-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-primary-bbf281-predictor-6b7cf7ddb4 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e81d6-predictor-6c8678b65-zmk7d |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-bbf281 |
UpdateFailed |
Failed to update status for InferenceService "isvc-primary-bbf281": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-bbf281": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-bbf281 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-bbf281": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e81d6-predictor-6c8678b65-zmk7d |
Killing |
Stopping container kserve-container | |
| (x8) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-e81d6-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-primary-bbf281-predictor-6b7cf7ddb4-9mhhw |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:72e9a53b4f6609f050c864ccf8bba3bc53b15bbbb0a09e5de74a3bdc7465dd84" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-primary-bbf281-predictor-6b7cf7ddb4-9mhhw |
AddedInterface |
Add eth0 [10.134.0.44/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-bbf281-predictor-6b7cf7ddb4-9mhhw |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-bbf281-predictor-6b7cf7ddb4-9mhhw |
Created |
Created container: storage-initializer | |
| (x17) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e81d6-predictor-6c8678b65-zmk7d |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.43:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-primary-bbf281-predictor-6b7cf7ddb4-9mhhw |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-bbf281-predictor-6b7cf7ddb4-9mhhw |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-bbf281-predictor-6b7cf7ddb4-9mhhw |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-bbf281-predictor-6b7cf7ddb4-9mhhw |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-bbf281-predictor-6b7cf7ddb4-9mhhw |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-bbf281-predictor-6b7cf7ddb4-9mhhw |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1435" already present on machine | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-e81d6-predictor-6c8678b65-zmk7d |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.43:8643/healthz": dial tcp 10.134.0.43:8643: connect: connection refused |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-bbf281-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-bbf281-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-primary-bbf281-predictor-6b7cf7ddb4-9mhhw |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.44:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-bbf281 |
InferenceServiceReady |
InferenceService [isvc-primary-bbf281] is Ready | |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-bbf281 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-bbf281 |
UpdateFailed |
Failed to update status for InferenceService "isvc-secondary-bbf281": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-bbf281": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-bbf281-predictor-7f76d5b559-s84pm |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-secondary-bbf281-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-secondary-bbf281-predictor-7f76d5b559 |
SuccessfulCreate |
Created pod: isvc-secondary-bbf281-predictor-7f76d5b559-s84pm | |
kserve-ci-e2e-test |
deployment-controller |
isvc-secondary-bbf281-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-secondary-bbf281-predictor-7f76d5b559 from 0 to 1 | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-bbf281 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-bbf281": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
multus |
isvc-secondary-bbf281-predictor-7f76d5b559-s84pm |
AddedInterface |
Add eth0 [10.134.0.45/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-bbf281-predictor-7f76d5b559-s84pm |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-bbf281-predictor-7f76d5b559-s84pm |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-bbf281-predictor-7f76d5b559-s84pm |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:72e9a53b4f6609f050c864ccf8bba3bc53b15bbbb0a09e5de74a3bdc7465dd84" already present on machine |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-bbf281-predictor-7f76d5b559-s84pm |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-secondary-bbf281-predictor-7f76d5b559-s84pm_kserve-ci-e2e-test(15cd8bda-e985-47ae-97fd-c320a2448ac4) | |
| (x14) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-bbf281 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-bbf281-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-bbf281-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-bbf281-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-bbf281-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
deployment-controller |
isvc-init-fail-8448e3-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-init-fail-8448e3-predictor-84676484d9 from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-init-fail-8448e3-predictor-84676484d9-mvhgn |
AddedInterface |
Add eth0 [10.134.0.46/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-8448e3 |
UpdateFailed |
Failed to update status for InferenceService "isvc-init-fail-8448e3": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-8448e3": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-8448e3 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-8448e3": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-bbf281-predictor-6b7cf7ddb4-9mhhw |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-bbf281-predictor-6b7cf7ddb4-9mhhw |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-init-fail-8448e3-predictor-84676484d9 |
SuccessfulCreate |
Created pod: isvc-init-fail-8448e3-predictor-84676484d9-mvhgn | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-bbf281-predictor-6b7cf7ddb4-9mhhw |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.44:8643/healthz": dial tcp 10.134.0.44:8643: connect: connection refused | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-8448e3-predictor-84676484d9-mvhgn |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-8448e3-predictor-84676484d9-mvhgn |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-8448e3-predictor-84676484d9-mvhgn |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:72e9a53b4f6609f050c864ccf8bba3bc53b15bbbb0a09e5de74a3bdc7465dd84" already present on machine |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-8448e3-predictor-84676484d9-mvhgn |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-init-fail-8448e3-predictor-84676484d9-mvhgn_kserve-ci-e2e-test(f0339805-0726-40e0-829f-34ba5030726d) | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-8448e3 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-926cf-predictor-6fc8899c49-49qgj |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:72e9a53b4f6609f050c864ccf8bba3bc53b15bbbb0a09e5de74a3bdc7465dd84" already present on machine | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-926cf-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-926cf-predictor-6fc8899c49 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-926cf-predictor-6fc8899c49-49qgj |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-926cf-predictor-6fc8899c49 |
SuccessfulCreate |
Created pod: raw-sklearn-926cf-predictor-6fc8899c49-49qgj | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-926cf-predictor-6fc8899c49-49qgj |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
raw-sklearn-926cf-predictor-6fc8899c49-49qgj |
AddedInterface |
Add eth0 [10.134.0.47/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-926cf |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-926cf": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-926cf": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-926cf |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-926cf": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-926cf-predictor-6fc8899c49-49qgj |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-926cf-predictor-6fc8899c49-49qgj |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-926cf-predictor-6fc8899c49-49qgj |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-926cf-predictor-6fc8899c49-49qgj |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-926cf-predictor-6fc8899c49-49qgj |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-926cf-predictor-6fc8899c49-49qgj |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1435" already present on machine | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-926cf-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-926cf-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-926cf |
InferenceServiceReady |
InferenceService [raw-sklearn-926cf] is Ready | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-926cf |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-926cf-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-926cf-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-926cf-predictor-6fc8899c49-49qgj |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-runtime-bca52-predictor-7cc55b85d5 |
SuccessfulCreate |
Created pod: raw-sklearn-runtime-bca52-predictor-7cc55b85d5-nl97d | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-926cf-predictor-6fc8899c49-49qgj |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-bca52 |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-runtime-bca52": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-bca52": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-runtime-bca52-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-runtime-bca52-predictor-7cc55b85d5 from 0 to 1 | |
kserve-ci-e2e-test |
multus |
raw-sklearn-runtime-bca52-predictor-7cc55b85d5-nl97d |
AddedInterface |
Add eth0 [10.134.0.48/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bca52-predictor-7cc55b85d5-nl97d |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:72e9a53b4f6609f050c864ccf8bba3bc53b15bbbb0a09e5de74a3bdc7465dd84" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bca52-predictor-7cc55b85d5-nl97d |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bca52-predictor-7cc55b85d5-nl97d |
Started |
Started container storage-initializer | |
| (x9) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-926cf-predictor-6fc8899c49-49qgj |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.47:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-926cf-predictor-6fc8899c49-49qgj |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.47:8643/healthz": dial tcp 10.134.0.47:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bca52-predictor-7cc55b85d5-nl97d |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bca52-predictor-7cc55b85d5-nl97d |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bca52-predictor-7cc55b85d5-nl97d |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bca52-predictor-7cc55b85d5-nl97d |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bca52-predictor-7cc55b85d5-nl97d |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bca52-predictor-7cc55b85d5-nl97d |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1435" already present on machine | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-bca52-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-bca52-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-bca52 |
InferenceServiceReady |
InferenceService [raw-sklearn-runtime-bca52] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-bca52 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-bca52-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-bca52-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bca52-predictor-7cc55b85d5-nl97d |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bca52-predictor-7cc55b85d5-nl97d |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bca52-predictor-7cc55b85d5-nl97d |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.48:8643/healthz": dial tcp 10.134.0.48:8643: connect: connection refused | |
| (x9) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-bca52-predictor-7cc55b85d5-nl97d |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.48:8080: connect: connection refused |