| Time | Namespace | Component | RelatedObject | Reason | Message |
|---|---|---|---|---|---|
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz to ip-10-0-143-50.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls to ip-10-0-143-50.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-b55a9-predictor-5b69cff558-tnkqz |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-b55a9-predictor-5b69cff558-tnkqz to ip-10-0-143-50.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-hpa-31388-7fb588c955-rf75t |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-hpa-31388-7fb588c955-rf75t to ip-10-0-131-235.ec2.internal | ||
kserve-ci-e2e-test |
isvc-secondary-771967-predictor-85df677c64-w8fhw |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-secondary-771967-predictor-85df677c64-w8fhw to ip-10-0-143-50.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4 to ip-10-0-143-50.ec2.internal | ||
kserve-ci-e2e-test |
isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn to ip-10-0-143-50.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7 to ip-10-0-143-50.ec2.internal | ||
kserve-ci-e2e-test |
isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg to ip-10-0-143-50.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-54928-7944f78864-9qnp4 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-54928-7944f78864-9qnp4 to ip-10-0-131-235.ec2.internal | ||
kserve-ci-e2e-test |
isvc-primary-771967-predictor-568fd899c9-8lrrm |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-primary-771967-predictor-568fd899c9-8lrrm to ip-10-0-143-50.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2 to ip-10-0-143-50.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779 to ip-10-0-143-50.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7 to ip-10-0-143-50.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-hpa-31388-predictor-57fb4985bf-rg7k4 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-31388-predictor-57fb4985bf-rg7k4 to ip-10-0-142-179.ec2.internal | ||
kserve-ci-e2e-test |
replicaset-controller |
isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd |
SuccessfulCreate |
Created pod: isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn | |
kserve-ci-e2e-test |
deployment-controller |
isvc-raw-sklearn-batcher-b8262-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd from 0 to 1 | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-b8262 |
UpdateFailed |
Failed to update status for InferenceService "isvc-raw-sklearn-batcher-b8262": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-b8262": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-b8262 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-b8262": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
multus |
isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn |
AddedInterface |
Add eth0 [10.132.0.20/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:359d904aaa602365d470b7fafd3828b888fb4c1e71dc1473bbad336c728c7088" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:359d904aaa602365d470b7fafd3828b888fb4c1e71dc1473bbad336c728c7088" in 3.547s (3.547s including waiting). Image size: 300567509 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1690" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1690" in 14.457s (14.457s including waiting). Image size: 1558595510 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.621s (2.621s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn |
Pulling |
Pulling image "quay.io/opendatahub/kserve-agent@sha256:694a44c3a2aed22c78d433e9220113d47feab5895f86a8ff0e64d7345c386ab0" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:694a44c3a2aed22c78d433e9220113d47feab5895f86a8ff0e64d7345c386ab0" in 2.517s (2.517s including waiting). Image size: 239994492 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn |
Started |
Started container agent | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-b8262-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-b8262-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-b8262 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-b8262 |
InferenceServiceReady |
InferenceService [isvc-raw-sklearn-batcher-b8262] is Ready | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-b8262-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-b8262-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-54928-predictor-5894b4f55d |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:359d904aaa602365d470b7fafd3828b888fb4c1e71dc1473bbad336c728c7088" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-54928-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-54928 |
UpdateFailed |
Failed to update status for InferenceService "isvc-xgboost-graph-raw-54928": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-graph-raw-54928": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7 |
AddedInterface |
Add eth0 [10.132.0.21/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-54928 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-54928": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-54928": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz |
AddedInterface |
Add eth0 [10.132.0.22/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:359d904aaa602365d470b7fafd3828b888fb4c1e71dc1473bbad336c728c7088" already present on machine | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-54928-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-54928-predictor-5894b4f55d from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1690" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.20:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-54928-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-54928-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x4) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-b8262-predictor-676f589bcd-6dpfn |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.20:8643/healthz": dial tcp 10.132.0.20:8643: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 17.996s (17.996s including waiting). Image size: 1305293052 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz |
Created |
Created container: kube-rbac-proxy | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-54928-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-54928-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-54928 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-54928 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-54928] is Ready | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-54928 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-54928 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-54928] is Ready | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-54928 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-54928] is Ready | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-54928-7944f78864 |
SuccessfulCreate |
Created pod: model-chainer-raw-54928-7944f78864-9qnp4 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-54928-7944f78864-9qnp4 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-router@sha256:739b6aa792fe51f6eef0c708781538ef463424156c64441d55ba532852ef9645" | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-54928-7944f78864-9qnp4 |
AddedInterface |
Add eth0 [10.134.0.35/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-54928 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-54928-7944f78864 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-54928-7944f78864-9qnp4 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:739b6aa792fe51f6eef0c708781538ef463424156c64441d55ba532852ef9645" in 2.78s (2.78s including waiting). Image size: 218070141 bytes. | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-54928-7944f78864-9qnp4 |
Started |
Started container model-chainer-raw-54928 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-54928-7944f78864-9qnp4 |
Created |
Created container: model-chainer-raw-54928 | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-54928-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-54928-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x6) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-54928-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x6) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-54928-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-31388-predictor-57fb4985bf-rg7k4 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:359d904aaa602365d470b7fafd3828b888fb4c1e71dc1473bbad336c728c7088" already present on machine | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-hpa-31388-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-hpa-31388-predictor-57fb4985bf from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-54928-7944f78864-9qnp4 |
Killing |
Stopping container model-chainer-raw-54928 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-31388 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-hpa-31388": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-hpa-31388": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-graph-raw-hpa-31388-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-hpa-31388-predictor-57fb4985bf-rg7k4 |
AddedInterface |
Add eth0 [10.133.0.20/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-hpa-31388-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-31388 |
UpdateFailed |
Failed to update status for InferenceService "isvc-xgboost-graph-raw-hpa-31388": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-xgboost-graph-raw-hpa-31388": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-31388-predictor-57fb4985bf-rg7k4 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-hpa-31388-predictor-57fb4985bf |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-hpa-31388-predictor-57fb4985bf-rg7k4 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-31388-predictor-57fb4985bf-rg7k4 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:359d904aaa602365d470b7fafd3828b888fb4c1e71dc1473bbad336c728c7088" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7 |
AddedInterface |
Add eth0 [10.132.0.23/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.22:8643/healthz": dial tcp 10.132.0.22:8643: connect: connection refused | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-54928-predictor-7cbb6588b5-n2wpz |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.22:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.21:8643/healthz": dial tcp 10.132.0.21:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-54928-predictor-5894b4f55d-bp6n7 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.21:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-31388-predictor-57fb4985bf-rg7k4 |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1690" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-31388-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-31388-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-31388-predictor-57fb4985bf-rg7k4 |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-31388-predictor-57fb4985bf-rg7k4 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-31388-predictor-57fb4985bf-rg7k4 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-31388-predictor-57fb4985bf-rg7k4 |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 18.698s (18.698s including waiting). Image size: 1305293052 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-31388-predictor-57fb4985bf-rg7k4 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-31388-predictor-57fb4985bf-rg7k4 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-31388-predictor-57fb4985bf-rg7k4 |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 3.072s (3.072s including waiting). Image size: 211946088 bytes. | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-54928-7944f78864-9qnp4 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-31388-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-31388-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-31388 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-31388 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-hpa-31388] is Ready | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-31388 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-hpa-31388] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-31388 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-31388-7fb588c955-rf75t |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-raw-hpa-31388-serving-cert" not found | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-31388 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-hpa-31388] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-hpa-31388 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-hpa-31388-7fb588c955 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-hpa-31388-7fb588c955 |
SuccessfulCreate |
Created pod: model-chainer-raw-hpa-31388-7fb588c955-rf75t | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-31388-7fb588c955-rf75t |
Started |
Started container model-chainer-raw-hpa-31388 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-31388-7fb588c955-rf75t |
Created |
Created container: model-chainer-raw-hpa-31388 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-31388-7fb588c955-rf75t |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:739b6aa792fe51f6eef0c708781538ef463424156c64441d55ba532852ef9645" already present on machine | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-hpa-31388-7fb588c955-rf75t |
AddedInterface |
Add eth0 [10.134.0.36/23] from ovn-kubernetes | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-31388-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x6) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-31388-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-31388-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x6) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-31388-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-31388-7fb588c955-rf75t |
Killing |
Stopping container model-chainer-raw-hpa-31388 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-31388-predictor-57fb4985bf-rg7k4 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-9f9d5-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-9f9d5-predictor-6c9bc64f49 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-9f9d5 |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-9f9d5": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-9f9d5": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-9f9d5-predictor-6c9bc64f49 |
SuccessfulCreate |
Created pod: message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-31388-predictor-57fb4985bf-rg7k4 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "message-dumper-raw-9f9d5-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4 |
AddedInterface |
Add eth0 [10.132.0.24/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4 |
Pulling |
Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4 |
Pulled |
Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.268s (1.268s including waiting). Image size: 14813193 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-31388-predictor-57fb4985bf-rg7k4 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.20:8643/healthz": dial tcp 10.133.0.20:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4 |
Started |
Started container kube-rbac-proxy | |
| (x8) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-31388-predictor-57fb4985bf-rg7k4 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.20:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.23:8643/healthz": dial tcp 10.132.0.23:8643: connect: connection refused | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-31388-predictor-9c6fd7d58-6f4r7 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.23:8080: connect: connection refused |
| (x8) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-9f9d5 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-9f9d5 |
InferenceServiceReady |
InferenceService [message-dumper-raw-9f9d5] is Ready | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-9f9d5 |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-9f9d5": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-9f9d5": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-9f9d5-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-9f9d5-predictor-5968dfc74b from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-9f9d5-predictor-5968dfc74b |
SuccessfulCreate |
Created pod: isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-9f9d5 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-9f9d5": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779 |
AddedInterface |
Add eth0 [10.132.0.25/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:359d904aaa602365d470b7fafd3828b888fb4c1e71dc1473bbad336c728c7088" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779 |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:694a44c3a2aed22c78d433e9220113d47feab5895f86a8ff0e64d7345c386ab0" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779 |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1690" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779 |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779 |
Created |
Created container: kube-rbac-proxy | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-31388-7fb588c955-rf75t |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-9f9d5-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-9f9d5-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-9f9d5-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-9f9d5-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-9f9d5-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-9f9d5-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-9f9d5 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-9f9d5 |
InferenceServiceReady |
InferenceService [isvc-logger-raw-9f9d5] is Ready | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-9f9d5-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-9f9d5-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-scale-raw-5a351-predictor-764cdcb448 |
SuccessfulCreate |
Created pod: isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-scale-raw-5a351-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-scale-raw-5a351-predictor-764cdcb448 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-5a351 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-scale-raw-5a351": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-scale-raw-5a351": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-9f9d5-predictor-6c9bc64f49-dk8q4 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2 |
AddedInterface |
Add eth0 [10.132.0.26/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:359d904aaa602365d470b7fafd3828b888fb4c1e71dc1473bbad336c728c7088" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779 |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1690" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2 |
Started |
Started container kube-rbac-proxy | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.25:8080: connect: connection refused |
| (x4) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.25:8643/healthz": dial tcp 10.132.0.25:8643: connect: connection refused |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-9f9d5-predictor-5968dfc74b-fg779 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-5a351-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-5a351-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x8) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-5a351-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x8) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-5a351-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-5a351 |
InferenceServiceReady |
InferenceService [isvc-sklearn-scale-raw-5a351] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-5a351 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-primary-771967-predictor-568fd899c9 |
SuccessfulCreate |
Created pod: isvc-primary-771967-predictor-568fd899c9-8lrrm | |
kserve-ci-e2e-test |
deployment-controller |
isvc-primary-771967-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-primary-771967-predictor-568fd899c9 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-771967-predictor-568fd899c9-8lrrm |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-primary-771967-predictor-serving-cert" not found | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-771967 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-771967": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-771967 |
UpdateFailed |
Failed to update status for InferenceService "isvc-primary-771967": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-771967": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-primary-771967-predictor-568fd899c9-8lrrm |
Started |
Started container storage-initializer | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.26:8643/healthz": dial tcp 10.132.0.26:8643: connect: connection refused |
kserve-ci-e2e-test |
multus |
isvc-primary-771967-predictor-568fd899c9-8lrrm |
AddedInterface |
Add eth0 [10.132.0.27/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-771967-predictor-568fd899c9-8lrrm |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:359d904aaa602365d470b7fafd3828b888fb4c1e71dc1473bbad336c728c7088" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-771967-predictor-568fd899c9-8lrrm |
Created |
Created container: storage-initializer | |
| (x18) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-5a351-predictor-764cdcb448-j4cp2 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.26:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-primary-771967-predictor-568fd899c9-8lrrm |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1690" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-771967-predictor-568fd899c9-8lrrm |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-771967-predictor-568fd899c9-8lrrm |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-771967-predictor-568fd899c9-8lrrm |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-771967-predictor-568fd899c9-8lrrm |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-771967-predictor-568fd899c9-8lrrm |
Started |
Started container kube-rbac-proxy | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-771967-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-771967-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-primary-771967-predictor-568fd899c9-8lrrm |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.27:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-771967 |
InferenceServiceReady |
InferenceService [isvc-primary-771967] is Ready | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-771967 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-771967 |
UpdateFailed |
Failed to update status for InferenceService "isvc-secondary-771967": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-771967": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-secondary-771967-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-secondary-771967-predictor-85df677c64 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-771967 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-771967": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-secondary-771967-predictor-85df677c64 |
SuccessfulCreate |
Created pod: isvc-secondary-771967-predictor-85df677c64-w8fhw | |
kserve-ci-e2e-test |
multus |
isvc-secondary-771967-predictor-85df677c64-w8fhw |
AddedInterface |
Add eth0 [10.132.0.28/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-771967-predictor-85df677c64-w8fhw |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:359d904aaa602365d470b7fafd3828b888fb4c1e71dc1473bbad336c728c7088" already present on machine |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-771967-predictor-85df677c64-w8fhw |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-771967-predictor-85df677c64-w8fhw |
Started |
Started container storage-initializer |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-771967-predictor-85df677c64-w8fhw |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-secondary-771967-predictor-85df677c64-w8fhw_kserve-ci-e2e-test(eac18698-274f-479b-b3f0-4e896c1e1211) | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-771967 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-771967-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-771967-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-771967-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-771967-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-primary-771967-predictor-568fd899c9-8lrrm |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-771967-predictor-568fd899c9-8lrrm |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-771967-predictor-568fd899c9-8lrrm |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.27:8643/healthz": dial tcp 10.132.0.27:8643: connect: connection refused | |
kserve-ci-e2e-test |
deployment-controller |
isvc-init-fail-e38b18-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-init-fail-e38b18-predictor-55d4c96d56 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-init-fail-e38b18-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-init-fail-e38b18-predictor-55d4c96d56 |
SuccessfulCreate |
Created pod: isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-e38b18 |
UpdateFailed |
Failed to update status for InferenceService "isvc-init-fail-e38b18": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-e38b18": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-e38b18 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-e38b18": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
multus |
isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg |
AddedInterface |
Add eth0 [10.132.0.29/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:359d904aaa602365d470b7fafd3828b888fb4c1e71dc1473bbad336c728c7088" already present on machine |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg |
Created |
Created container: storage-initializer |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-e38b18 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-e38b18-predictor-55d4c96d56-rtvmg |
Killing |
Stopping container storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-b55a9-predictor-5b69cff558 |
SuccessfulCreate |
Created pod: raw-sklearn-b55a9-predictor-5b69cff558-tnkqz | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-b55a9 |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-b55a9": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-b55a9": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-b55a9-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-b55a9-predictor-5b69cff558 from 0 to 1 | |
kserve-ci-e2e-test |
multus |
raw-sklearn-b55a9-predictor-5b69cff558-tnkqz |
AddedInterface |
Add eth0 [10.132.0.30/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-b55a9-predictor-5b69cff558-tnkqz |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:359d904aaa602365d470b7fafd3828b888fb4c1e71dc1473bbad336c728c7088" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-b55a9-predictor-5b69cff558-tnkqz |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-b55a9-predictor-5b69cff558-tnkqz |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-b55a9-predictor-5b69cff558-tnkqz |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1690" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-b55a9-predictor-5b69cff558-tnkqz |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-b55a9-predictor-5b69cff558-tnkqz |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-b55a9-predictor-5b69cff558-tnkqz |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-b55a9-predictor-5b69cff558-tnkqz |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-b55a9-predictor-5b69cff558-tnkqz |
Created |
Created container: kube-rbac-proxy | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-b55a9-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-b55a9-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-b55a9 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-b55a9 |
InferenceServiceReady |
InferenceService [raw-sklearn-b55a9] is Ready | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-b55a9-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-b55a9-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-b55a9-predictor-5b69cff558-tnkqz |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-b55a9-predictor-5b69cff558-tnkqz |
Killing |
Stopping container kserve-container | |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-b55a9-predictor-5b69cff558-tnkqz |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.30:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-b55a9-predictor-5b69cff558-tnkqz |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.30:8643/healthz": dial tcp 10.132.0.30:8643: connect: connection refused | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-runtime-68197-predictor-6d5bc8f86b |
SuccessfulCreate |
Created pod: raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:359d904aaa602365d470b7fafd3828b888fb4c1e71dc1473bbad336c728c7088" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls |
AddedInterface |
Add eth0 [10.132.0.31/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-68197 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-68197": the object has been modified; please apply your changes to the latest version and try again | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-68197 |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-runtime-68197": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-68197": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-runtime-68197-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-runtime-68197-predictor-6d5bc8f86b from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1690" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-68197-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-68197-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-68197 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-68197 |
InferenceServiceReady |
InferenceService [raw-sklearn-runtime-68197] is Ready | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls |
Killing |
Stopping container kube-rbac-proxy | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-68197-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-68197-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls |
Killing |
Stopping container kserve-container | |
| (x11) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.31:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-68197-predictor-6d5bc8f86b-fxhls |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.31:8643/healthz": dial tcp 10.132.0.31:8643: connect: connection refused |