| Time | Namespace | Component | RelatedObject | Reason | Message |
|---|---|---|---|---|---|
e2e-gw1-20542c80ea |
success-200-isvc-ff374-predictor-5687f8548d-2mkf8 |
Scheduled |
Successfully assigned e2e-gw1-20542c80ea/success-200-isvc-ff374-predictor-5687f8548d-2mkf8 to ip-10-0-131-204.us-east-2.compute.internal | ||
e2e-gw1-20542c80ea |
success-200-isvc-f04bb-predictor-5c849f7f88-rncqq |
Scheduled |
Successfully assigned e2e-gw1-20542c80ea/success-200-isvc-f04bb-predictor-5c849f7f88-rncqq to ip-10-0-131-204.us-east-2.compute.internal | ||
e2e-gw1-20542c80ea |
error-404-isvc-2794d-predictor-579f6cb57-c77lq |
Scheduled |
Successfully assigned e2e-gw1-20542c80ea/error-404-isvc-2794d-predictor-579f6cb57-c77lq to ip-10-0-142-227.us-east-2.compute.internal | ||
e2e-gw1-20542c80ea |
error-404-isvc-c85ae-predictor-6db4cb9c4f-gr5x9 |
Scheduled |
Successfully assigned e2e-gw1-20542c80ea/error-404-isvc-c85ae-predictor-6db4cb9c4f-gr5x9 to ip-10-0-142-227.us-east-2.compute.internal | ||
e2e-gw1-20542c80ea |
error-404-isvc-f04bb-predictor-5cb4948585-xcrsk |
Scheduled |
Successfully assigned e2e-gw1-20542c80ea/error-404-isvc-f04bb-predictor-5cb4948585-xcrsk to ip-10-0-142-227.us-east-2.compute.internal | ||
e2e-gw1-20542c80ea |
isvc-sklearn-graph-1-predictor-7d7f966fd5-6v9r7 |
Scheduled |
Successfully assigned e2e-gw1-20542c80ea/isvc-sklearn-graph-1-predictor-7d7f966fd5-6v9r7 to ip-10-0-138-133.us-east-2.compute.internal | ||
e2e-gw1-20542c80ea |
isvc-xgboost-graph-predictor-6b76945699-fkvcm |
Scheduled |
Successfully assigned e2e-gw1-20542c80ea/isvc-xgboost-graph-predictor-6b76945699-fkvcm to ip-10-0-138-133.us-east-2.compute.internal | ||
e2e-gw1-20542c80ea |
model-chainer-8468d97d58-f6qrf |
Scheduled |
Successfully assigned e2e-gw1-20542c80ea/model-chainer-8468d97d58-f6qrf to ip-10-0-131-204.us-east-2.compute.internal | ||
e2e-gw1-20542c80ea |
error-404-isvc-ff374-predictor-6cbb85d857-clcb4 |
Scheduled |
Successfully assigned e2e-gw1-20542c80ea/error-404-isvc-ff374-predictor-6cbb85d857-clcb4 to ip-10-0-142-227.us-east-2.compute.internal | ||
e2e-gw1-20542c80ea |
sequence-graph-ff374-57f4fd58f-qm6xd |
Scheduled |
Successfully assigned e2e-gw1-20542c80ea/sequence-graph-ff374-57f4fd58f-qm6xd to ip-10-0-131-204.us-east-2.compute.internal | ||
e2e-gw1-20542c80ea |
switch-graph-f04bb-85d9475c98-vrqtr |
Scheduled |
Successfully assigned e2e-gw1-20542c80ea/switch-graph-f04bb-85d9475c98-vrqtr to ip-10-0-138-133.us-east-2.compute.internal | ||
e2e-gw1-20542c80ea |
success-200-isvc-2794d-predictor-b4b4bc48-m57wl |
Scheduled |
Successfully assigned e2e-gw1-20542c80ea/success-200-isvc-2794d-predictor-b4b4bc48-m57wl to ip-10-0-131-204.us-east-2.compute.internal | ||
e2e-gw1-20542c80ea |
success-200-isvc-c85ae-predictor-74dd747476-m87h9 |
Scheduled |
Successfully assigned e2e-gw1-20542c80ea/success-200-isvc-c85ae-predictor-74dd747476-m87h9 to ip-10-0-131-204.us-east-2.compute.internal | ||
e2e-gw1-20542c80ea |
isvc-sklearn-graph-2-predictor-857f478b78-vqd89 |
Scheduled |
Successfully assigned e2e-gw1-20542c80ea/isvc-sklearn-graph-2-predictor-857f478b78-vqd89 to ip-10-0-138-133.us-east-2.compute.internal | ||
e2e-gw1-20542c80ea |
sequence-graph-2794d-566c984449-8vcgq |
Scheduled |
Successfully assigned e2e-gw1-20542c80ea/sequence-graph-2794d-566c984449-8vcgq to ip-10-0-138-133.us-east-2.compute.internal | ||
e2e-gw1-20542c80ea |
sequence-graph-c85ae-78bd9895bb-qx9sm |
Scheduled |
Successfully assigned e2e-gw1-20542c80ea/sequence-graph-c85ae-78bd9895bb-qx9sm to ip-10-0-138-133.us-east-2.compute.internal | ||
e2e-gw1-20542c80ea |
v1beta1Controllers |
isvc-sklearn-graph-1 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-1": the object has been modified; please apply your changes to the latest version and try again | |
e2e-gw1-20542c80ea |
deployment-controller |
isvc-sklearn-graph-1-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-1-predictor-7d7f966fd5 from 0 to 1 | |
e2e-gw1-20542c80ea |
v1beta1Controllers |
isvc-sklearn-graph-1 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-1": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-1": the object has been modified; please apply your changes to the latest version and try again | |
e2e-gw1-20542c80ea |
replicaset-controller |
isvc-sklearn-graph-1-predictor-7d7f966fd5 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-1-predictor-7d7f966fd5-6v9r7 | |
e2e-gw1-20542c80ea |
deployment-controller |
isvc-xgboost-graph-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-predictor-6b76945699 from 0 to 1 | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-2-predictor-857f478b78-vqd89 |
Started |
Container started | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-1-predictor-7d7f966fd5-6v9r7 |
Created |
Container created | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-2-predictor-857f478b78-vqd89 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d0b9b1a20c75100104d761661021a933899bdbf04932977a64e3c751be1a2b7" already present on machine and can be accessed by the pod | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-2-predictor-857f478b78-vqd89 |
Created |
Container created | |
e2e-gw1-20542c80ea |
replicaset-controller |
isvc-sklearn-graph-2-predictor-857f478b78 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-2-predictor-857f478b78-vqd89 | |
e2e-gw1-20542c80ea |
multus |
isvc-sklearn-graph-1-predictor-7d7f966fd5-6v9r7 |
AddedInterface |
Add eth0 [10.134.0.22/23] from ovn-kubernetes | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-1-predictor-7d7f966fd5-6v9r7 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d0b9b1a20c75100104d761661021a933899bdbf04932977a64e3c751be1a2b7" already present on machine and can be accessed by the pod | |
e2e-gw1-20542c80ea |
multus |
isvc-sklearn-graph-2-predictor-857f478b78-vqd89 |
AddedInterface |
Add eth0 [10.134.0.24/23] from ovn-kubernetes | |
e2e-gw1-20542c80ea |
kubelet |
isvc-xgboost-graph-predictor-6b76945699-fkvcm |
Started |
Container started | |
e2e-gw1-20542c80ea |
kubelet |
isvc-xgboost-graph-predictor-6b76945699-fkvcm |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:0d0b9b1a20c75100104d761661021a933899bdbf04932977a64e3c751be1a2b7" already present on machine and can be accessed by the pod | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-1-predictor-7d7f966fd5-6v9r7 |
Started |
Container started | |
e2e-gw1-20542c80ea |
replicaset-controller |
isvc-xgboost-graph-predictor-6b76945699 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-predictor-6b76945699-fkvcm | |
e2e-gw1-20542c80ea |
multus |
isvc-xgboost-graph-predictor-6b76945699-fkvcm |
AddedInterface |
Add eth0 [10.134.0.23/23] from ovn-kubernetes | |
e2e-gw1-20542c80ea |
deployment-controller |
isvc-sklearn-graph-2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-2-predictor-857f478b78 from 0 to 1 | |
e2e-gw1-20542c80ea |
kubelet |
isvc-xgboost-graph-predictor-6b76945699-fkvcm |
Created |
Container created | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-1-predictor-7d7f966fd5-6v9r7 |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-2147" | |
e2e-gw1-20542c80ea |
kubelet |
isvc-xgboost-graph-predictor-6b76945699-fkvcm |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-2-predictor-857f478b78-vqd89 |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-2147" | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-1-predictor-7d7f966fd5-6v9r7 |
Created |
Container created | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-1-predictor-7d7f966fd5-6v9r7 |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-2147" in 23.041s (23.041s including waiting). Image size: 1629853585 bytes. | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-1-predictor-7d7f966fd5-6v9r7 |
Started |
Container started | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-1-predictor-7d7f966fd5-6v9r7 |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-2-predictor-857f478b78-vqd89 |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-2-predictor-857f478b78-vqd89 |
Started |
Container started | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-2-predictor-857f478b78-vqd89 |
Created |
Container created | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-2-predictor-857f478b78-vqd89 |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-2147" in 23.881s (23.881s including waiting). Image size: 1629853585 bytes. | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-1-predictor-7d7f966fd5-6v9r7 |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" in 3.152s (3.152s including waiting). Image size: 184747123 bytes. | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-2-predictor-857f478b78-vqd89 |
Created |
Container created | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-1-predictor-7d7f966fd5-6v9r7 |
Created |
Container created | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-2-predictor-857f478b78-vqd89 |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" in 1.908s (1.908s including waiting). Image size: 184747123 bytes. | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-1-predictor-7d7f966fd5-6v9r7 |
Started |
Container started | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-2-predictor-857f478b78-vqd89 |
Started |
Container started | |
e2e-gw1-20542c80ea |
kubelet |
isvc-xgboost-graph-predictor-6b76945699-fkvcm |
Started |
Container started | |
e2e-gw1-20542c80ea |
kubelet |
isvc-xgboost-graph-predictor-6b76945699-fkvcm |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 28.558s (28.558s including waiting). Image size: 1460330245 bytes. | |
e2e-gw1-20542c80ea |
kubelet |
isvc-xgboost-graph-predictor-6b76945699-fkvcm |
Created |
Container created | |
e2e-gw1-20542c80ea |
kubelet |
isvc-xgboost-graph-predictor-6b76945699-fkvcm |
Started |
Container started | |
e2e-gw1-20542c80ea |
kubelet |
isvc-xgboost-graph-predictor-6b76945699-fkvcm |
Created |
Container created | |
e2e-gw1-20542c80ea |
kubelet |
isvc-xgboost-graph-predictor-6b76945699-fkvcm |
Pulled |
Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod | |
| (x3) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
isvc-xgboost-graph-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
isvc-xgboost-graph-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
isvc-sklearn-graph-1-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
isvc-sklearn-graph-1-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x5) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
isvc-sklearn-graph-2-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x5) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
isvc-sklearn-graph-2-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x8) | e2e-gw1-20542c80ea |
kubelet |
isvc-xgboost-graph-predictor-6b76945699-fkvcm |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.23:8080: connect: connection refused |
| (x9) | e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-1-predictor-7d7f966fd5-6v9r7 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.22:8080: connect: connection refused |
| (x9) | e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-2-predictor-857f478b78-vqd89 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.24:8080: connect: connection refused |
e2e-gw1-20542c80ea |
v1beta1Controllers |
isvc-xgboost-graph |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph] is Ready | |
| (x9) | e2e-gw1-20542c80ea |
v1beta1Controllers |
isvc-xgboost-graph |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
e2e-gw1-20542c80ea |
v1beta1Controllers |
isvc-sklearn-graph-2 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-2] is Ready | |
e2e-gw1-20542c80ea |
v1beta1Controllers |
isvc-sklearn-graph-1 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-1] is Ready | |
| (x9) | e2e-gw1-20542c80ea |
v1beta1Controllers |
isvc-sklearn-graph-2 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x9) | e2e-gw1-20542c80ea |
v1beta1Controllers |
isvc-sklearn-graph-1 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x5) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
isvc-xgboost-graph-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
isvc-xgboost-graph-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
isvc-sklearn-graph-2-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
isvc-sklearn-graph-2-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
e2e-gw1-20542c80ea |
InferenceGraphController |
model-chainer |
InternalError |
fails to update InferenceGraph status: Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "model-chainer": the object has been modified; please apply your changes to the latest version and try again | |
e2e-gw1-20542c80ea |
InferenceGraphController |
model-chainer |
UpdateFailed |
Failed to update status for InferenceGraph "model-chainer": Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "model-chainer": the object has been modified; please apply your changes to the latest version and try again | |
e2e-gw1-20542c80ea |
InferenceGraphController |
model-chainer |
InferenceGraphReady |
InferenceGraph [model-chainer] is Ready | |
e2e-gw1-20542c80ea |
deployment-controller |
model-chainer |
ScalingReplicaSet |
Scaled up replica set model-chainer-8468d97d58 from 0 to 1 | |
e2e-gw1-20542c80ea |
replicaset-controller |
model-chainer-8468d97d58 |
SuccessfulCreate |
Created pod: model-chainer-8468d97d58-f6qrf | |
e2e-gw1-20542c80ea |
kubelet |
model-chainer-8468d97d58-f6qrf |
Started |
Container started | |
e2e-gw1-20542c80ea |
kubelet |
model-chainer-8468d97d58-f6qrf |
Created |
Container created | |
e2e-gw1-20542c80ea |
kubelet |
model-chainer-8468d97d58-f6qrf |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:561002e842ac9bbf33141cfcc92eb7b7cc604d16ec35596c7ce81c9e22e080f4" already present on machine and can be accessed by the pod | |
e2e-gw1-20542c80ea |
multus |
model-chainer-8468d97d58-f6qrf |
AddedInterface |
Add eth0 [10.132.0.33/23] from ovn-kubernetes | |
| (x5) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
isvc-sklearn-graph-1-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
isvc-sklearn-graph-1-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
e2e-gw1-20542c80ea |
kubelet |
isvc-xgboost-graph-predictor-6b76945699-fkvcm |
Killing |
Stopping container kube-rbac-proxy | |
e2e-gw1-20542c80ea |
kubelet |
isvc-xgboost-graph-predictor-6b76945699-fkvcm |
Killing |
Stopping container kserve-container | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-2-predictor-857f478b78-vqd89 |
Killing |
Stopping container kserve-container | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-2-predictor-857f478b78-vqd89 |
Killing |
Stopping container kube-rbac-proxy | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-1-predictor-7d7f966fd5-6v9r7 |
Killing |
Stopping container kserve-container | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-1-predictor-7d7f966fd5-6v9r7 |
Killing |
Stopping container kube-rbac-proxy | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-2-predictor-857f478b78-vqd89 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.24:8643/healthz": dial tcp 10.134.0.24:8643: connect: connection refused | |
e2e-gw1-20542c80ea |
kubelet |
isvc-sklearn-graph-1-predictor-7d7f966fd5-6v9r7 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.22:8643/healthz": dial tcp 10.134.0.22:8643: connect: connection refused | |
e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
model-chainer |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
model-chainer |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
e2e-gw1-20542c80ea |
v1beta1Controllers |
success-200-isvc-ff374 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-ff374": the object has been modified; please apply your changes to the latest version and try again | |
e2e-gw1-20542c80ea |
v1beta1Controllers |
success-200-isvc-ff374 |
UpdateFailed |
Failed to update status for InferenceService "success-200-isvc-ff374": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-ff374": the object has been modified; please apply your changes to the latest version and try again | |
e2e-gw1-20542c80ea |
deployment-controller |
success-200-isvc-ff374-predictor |
ScalingReplicaSet |
Scaled up replica set success-200-isvc-ff374-predictor-5687f8548d from 0 to 1 | |
e2e-gw1-20542c80ea |
replicaset-controller |
success-200-isvc-ff374-predictor-5687f8548d |
SuccessfulCreate |
Created pod: success-200-isvc-ff374-predictor-5687f8548d-2mkf8 | |
e2e-gw1-20542c80ea |
deployment-controller |
error-404-isvc-ff374-predictor |
ScalingReplicaSet |
Scaled up replica set error-404-isvc-ff374-predictor-6cbb85d857 from 0 to 1 | |
e2e-gw1-20542c80ea |
multus |
success-200-isvc-ff374-predictor-5687f8548d-2mkf8 |
AddedInterface |
Add eth0 [10.132.0.34/23] from ovn-kubernetes | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-ff374-predictor-5687f8548d-2mkf8 |
Created |
Container created | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-ff374-predictor-6cbb85d857-clcb4 |
Created |
Container created | |
e2e-gw1-20542c80ea |
replicaset-controller |
error-404-isvc-ff374-predictor-6cbb85d857 |
SuccessfulCreate |
Created pod: error-404-isvc-ff374-predictor-6cbb85d857-clcb4 | |
e2e-gw1-20542c80ea |
multus |
error-404-isvc-ff374-predictor-6cbb85d857-clcb4 |
AddedInterface |
Add eth0 [10.133.0.27/23] from ovn-kubernetes | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-ff374-predictor-5687f8548d-2mkf8 |
Pulled |
Container image "quay.io/opendatahub/success-200-isvc:odh-pr-2147" already present on machine and can be accessed by the pod | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-ff374-predictor-5687f8548d-2mkf8 |
Created |
Container created | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-ff374-predictor-5687f8548d-2mkf8 |
Started |
Container started | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-ff374-predictor-5687f8548d-2mkf8 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-ff374-predictor-6cbb85d857-clcb4 |
Started |
Container started | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-ff374-predictor-5687f8548d-2mkf8 |
Started |
Container started | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-ff374-predictor-6cbb85d857-clcb4 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-ff374-predictor-6cbb85d857-clcb4 |
Started |
Container started | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-ff374-predictor-6cbb85d857-clcb4 |
Pulled |
Container image "quay.io/opendatahub/error-404-isvc:odh-pr-2147" already present on machine and can be accessed by the pod | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-ff374-predictor-6cbb85d857-clcb4 |
Created |
Container created | |
| (x2) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
success-200-isvc-ff374-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
success-200-isvc-ff374-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
error-404-isvc-ff374-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
error-404-isvc-ff374-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
e2e-gw1-20542c80ea |
v1beta1Controllers |
success-200-isvc-ff374 |
InferenceServiceReady |
InferenceService [success-200-isvc-ff374] is Ready | |
| (x5) | e2e-gw1-20542c80ea |
v1beta1Controllers |
success-200-isvc-ff374 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x5) | e2e-gw1-20542c80ea |
v1beta1Controllers |
error-404-isvc-ff374 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
e2e-gw1-20542c80ea |
v1beta1Controllers |
error-404-isvc-ff374 |
InferenceServiceReady |
InferenceService [error-404-isvc-ff374] is Ready | |
e2e-gw1-20542c80ea |
deployment-controller |
sequence-graph-ff374 |
ScalingReplicaSet |
Scaled up replica set sequence-graph-ff374-57f4fd58f from 0 to 1 | |
e2e-gw1-20542c80ea |
InferenceGraphController |
sequence-graph-ff374 |
InferenceGraphReady |
InferenceGraph [sequence-graph-ff374] is Ready | |
e2e-gw1-20542c80ea |
replicaset-controller |
sequence-graph-ff374-57f4fd58f |
SuccessfulCreate |
Created pod: sequence-graph-ff374-57f4fd58f-qm6xd | |
e2e-gw1-20542c80ea |
kubelet |
sequence-graph-ff374-57f4fd58f-qm6xd |
Created |
Container created | |
e2e-gw1-20542c80ea |
InferenceGraphController |
sequence-graph-ff374 |
InternalError |
fails to update InferenceGraph status: Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "sequence-graph-ff374": the object has been modified; please apply your changes to the latest version and try again | |
e2e-gw1-20542c80ea |
multus |
sequence-graph-ff374-57f4fd58f-qm6xd |
AddedInterface |
Add eth0 [10.132.0.36/23] from ovn-kubernetes | |
e2e-gw1-20542c80ea |
kubelet |
sequence-graph-ff374-57f4fd58f-qm6xd |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:561002e842ac9bbf33141cfcc92eb7b7cc604d16ec35596c7ce81c9e22e080f4" already present on machine and can be accessed by the pod | |
e2e-gw1-20542c80ea |
kubelet |
sequence-graph-ff374-57f4fd58f-qm6xd |
Started |
Container started | |
e2e-gw1-20542c80ea |
InferenceGraphController |
sequence-graph-ff374 |
UpdateFailed |
Failed to update status for InferenceGraph "sequence-graph-ff374": Operation cannot be fulfilled on inferencegraphs.serving.kserve.io "sequence-graph-ff374": the object has been modified; please apply your changes to the latest version and try again | |
| (x3) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
error-404-isvc-ff374-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
error-404-isvc-ff374-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
sequence-graph-ff374 |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
sequence-graph-ff374 |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x4) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
success-200-isvc-ff374-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
success-200-isvc-ff374-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-ff374-predictor-5687f8548d-2mkf8 |
Killing |
Stopping container kube-rbac-proxy | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-ff374-predictor-6cbb85d857-clcb4 |
Killing |
Stopping container kube-rbac-proxy | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-ff374-predictor-6cbb85d857-clcb4 |
Killing |
Stopping container kserve-container | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-ff374-predictor-5687f8548d-2mkf8 |
Killing |
Stopping container kserve-container | |
| (x8) | e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-ff374-predictor-5687f8548d-2mkf8 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.34:8080: connect: connection refused |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-ff374-predictor-5687f8548d-2mkf8 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.34:8643/healthz": dial tcp 10.132.0.34:8643: connect: connection refused | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-ff374-predictor-6cbb85d857-clcb4 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.27:8643/healthz": dial tcp 10.133.0.27:8643: connect: connection refused | |
| (x8) | e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-ff374-predictor-6cbb85d857-clcb4 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.27:8080: connect: connection refused |
e2e-gw1-20542c80ea |
replicaset-controller |
error-404-isvc-2794d-predictor-579f6cb57 |
SuccessfulCreate |
Created pod: error-404-isvc-2794d-predictor-579f6cb57-c77lq | |
e2e-gw1-20542c80ea |
v1beta1Controllers |
success-200-isvc-2794d |
UpdateFailed |
Failed to update status for InferenceService "success-200-isvc-2794d": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-2794d": the object has been modified; please apply your changes to the latest version and try again | |
e2e-gw1-20542c80ea |
deployment-controller |
error-404-isvc-2794d-predictor |
ScalingReplicaSet |
Scaled up replica set error-404-isvc-2794d-predictor-579f6cb57 from 0 to 1 | |
e2e-gw1-20542c80ea |
v1beta1Controllers |
success-200-isvc-2794d |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-2794d": the object has been modified; please apply your changes to the latest version and try again | |
e2e-gw1-20542c80ea |
multus |
success-200-isvc-2794d-predictor-b4b4bc48-m57wl |
AddedInterface |
Add eth0 [10.132.0.38/23] from ovn-kubernetes | |
e2e-gw1-20542c80ea |
deployment-controller |
success-200-isvc-2794d-predictor |
ScalingReplicaSet |
Scaled up replica set success-200-isvc-2794d-predictor-b4b4bc48 from 0 to 1 | |
e2e-gw1-20542c80ea |
replicaset-controller |
success-200-isvc-2794d-predictor-b4b4bc48 |
SuccessfulCreate |
Created pod: success-200-isvc-2794d-predictor-b4b4bc48-m57wl | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-2794d-predictor-b4b4bc48-m57wl |
Pulled |
Container image "quay.io/opendatahub/success-200-isvc:odh-pr-2147" already present on machine and can be accessed by the pod | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-2794d-predictor-b4b4bc48-m57wl |
Created |
Container created | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-2794d-predictor-579f6cb57-c77lq |
Started |
Container started | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-2794d-predictor-579f6cb57-c77lq |
Pulled |
Container image "quay.io/opendatahub/error-404-isvc:odh-pr-2147" already present on machine and can be accessed by the pod | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-2794d-predictor-b4b4bc48-m57wl |
Started |
Container started | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-2794d-predictor-b4b4bc48-m57wl |
Created |
Container created | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-2794d-predictor-b4b4bc48-m57wl |
Pulled |
Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-2794d-predictor-579f6cb57-c77lq |
Created |
Container created | |
e2e-gw1-20542c80ea |
multus |
error-404-isvc-2794d-predictor-579f6cb57-c77lq |
AddedInterface |
Add eth0 [10.133.0.29/23] from ovn-kubernetes | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-2794d-predictor-579f6cb57-c77lq |
Started |
Container started | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-2794d-predictor-579f6cb57-c77lq |
Pulled |
Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-2794d-predictor-b4b4bc48-m57wl |
Started |
Container started | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-2794d-predictor-579f6cb57-c77lq |
Created |
Container created | |
e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
success-200-isvc-2794d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
success-200-isvc-2794d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x2) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
error-404-isvc-2794d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
error-404-isvc-2794d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x7) | e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-2794d-predictor-579f6cb57-c77lq |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.29:8080: connect: connection refused |
| (x7) | e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-2794d-predictor-b4b4bc48-m57wl |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.38:8080: connect: connection refused |
| (x5) | e2e-gw1-20542c80ea |
v1beta1Controllers |
error-404-isvc-2794d |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
e2e-gw1-20542c80ea |
v1beta1Controllers |
error-404-isvc-2794d |
InferenceServiceReady |
InferenceService [error-404-isvc-2794d] is Ready | |
e2e-gw1-20542c80ea |
v1beta1Controllers |
success-200-isvc-2794d |
InferenceServiceReady |
InferenceService [success-200-isvc-2794d] is Ready | |
| (x5) | e2e-gw1-20542c80ea |
v1beta1Controllers |
success-200-isvc-2794d |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
e2e-gw1-20542c80ea |
deployment-controller |
sequence-graph-2794d |
ScalingReplicaSet |
Scaled up replica set sequence-graph-2794d-566c984449 from 0 to 1 | |
e2e-gw1-20542c80ea |
replicaset-controller |
sequence-graph-2794d-566c984449 |
SuccessfulCreate |
Created pod: sequence-graph-2794d-566c984449-8vcgq | |
e2e-gw1-20542c80ea |
InferenceGraphController |
sequence-graph-2794d |
InferenceGraphReady |
InferenceGraph [sequence-graph-2794d] is Ready | |
e2e-gw1-20542c80ea |
kubelet |
sequence-graph-2794d-566c984449-8vcgq |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:561002e842ac9bbf33141cfcc92eb7b7cc604d16ec35596c7ce81c9e22e080f4" already present on machine and can be accessed by the pod | |
e2e-gw1-20542c80ea |
kubelet |
sequence-graph-2794d-566c984449-8vcgq |
Started |
Container started | |
e2e-gw1-20542c80ea |
kubelet |
sequence-graph-2794d-566c984449-8vcgq |
Created |
Container created | |
e2e-gw1-20542c80ea |
multus |
sequence-graph-2794d-566c984449-8vcgq |
AddedInterface |
Add eth0 [10.134.0.26/23] from ovn-kubernetes | |
| (x3) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
error-404-isvc-2794d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
success-200-isvc-2794d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
success-200-isvc-2794d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
error-404-isvc-2794d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-2794d-predictor-579f6cb57-c77lq |
Killing |
Stopping container kube-rbac-proxy | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-2794d-predictor-b4b4bc48-m57wl |
Killing |
Stopping container kserve-container | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-2794d-predictor-b4b4bc48-m57wl |
Killing |
Stopping container kube-rbac-proxy | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-2794d-predictor-579f6cb57-c77lq |
Killing |
Stopping container kserve-container | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-2794d-predictor-b4b4bc48-m57wl |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.38:8643/healthz": dial tcp 10.132.0.38:8643: connect: connection refused | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-2794d-predictor-579f6cb57-c77lq |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.29:8643/healthz": dial tcp 10.133.0.29:8643: connect: connection refused | |
e2e-gw1-20542c80ea |
replicaset-controller |
error-404-isvc-c85ae-predictor-6db4cb9c4f |
SuccessfulCreate |
Created pod: error-404-isvc-c85ae-predictor-6db4cb9c4f-gr5x9 | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-c85ae-predictor-74dd747476-m87h9 |
Started |
Container started | |
e2e-gw1-20542c80ea |
v1beta1Controllers |
success-200-isvc-c85ae |
UpdateFailed |
Failed to update status for InferenceService "success-200-isvc-c85ae": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-c85ae": the object has been modified; please apply your changes to the latest version and try again | |
e2e-gw1-20542c80ea |
v1beta1Controllers |
success-200-isvc-c85ae |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-c85ae": the object has been modified; please apply your changes to the latest version and try again | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-c85ae-predictor-74dd747476-m87h9 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-c85ae-predictor-74dd747476-m87h9 |
Started |
Container started | |
e2e-gw1-20542c80ea |
multus |
success-200-isvc-c85ae-predictor-74dd747476-m87h9 |
AddedInterface |
Add eth0 [10.132.0.40/23] from ovn-kubernetes | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-c85ae-predictor-74dd747476-m87h9 |
Pulled |
Container image "quay.io/opendatahub/success-200-isvc:odh-pr-2147" already present on machine and can be accessed by the pod | |
e2e-gw1-20542c80ea |
deployment-controller |
error-404-isvc-c85ae-predictor |
ScalingReplicaSet |
Scaled up replica set error-404-isvc-c85ae-predictor-6db4cb9c4f from 0 to 1 | |
e2e-gw1-20542c80ea |
deployment-controller |
success-200-isvc-c85ae-predictor |
ScalingReplicaSet |
Scaled up replica set success-200-isvc-c85ae-predictor-74dd747476 from 0 to 1 | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-c85ae-predictor-74dd747476-m87h9 |
Created |
Container created | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-c85ae-predictor-74dd747476-m87h9 |
Created |
Container created | |
e2e-gw1-20542c80ea |
replicaset-controller |
success-200-isvc-c85ae-predictor-74dd747476 |
SuccessfulCreate |
Created pod: success-200-isvc-c85ae-predictor-74dd747476-m87h9 | |
e2e-gw1-20542c80ea |
multus |
error-404-isvc-c85ae-predictor-6db4cb9c4f-gr5x9 |
AddedInterface |
Add eth0 [10.133.0.31/23] from ovn-kubernetes | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-c85ae-predictor-6db4cb9c4f-gr5x9 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-c85ae-predictor-6db4cb9c4f-gr5x9 |
Created |
Container created | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-c85ae-predictor-6db4cb9c4f-gr5x9 |
Started |
Container started | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-c85ae-predictor-6db4cb9c4f-gr5x9 |
Started |
Container started | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-c85ae-predictor-6db4cb9c4f-gr5x9 |
Created |
Container created | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-c85ae-predictor-6db4cb9c4f-gr5x9 |
Pulled |
Container image "quay.io/opendatahub/error-404-isvc:odh-pr-2147" already present on machine and can be accessed by the pod | |
| (x2) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
sequence-graph-2794d |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
sequence-graph-2794d |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
error-404-isvc-c85ae-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
error-404-isvc-c85ae-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
success-200-isvc-c85ae-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
success-200-isvc-c85ae-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x5) | e2e-gw1-20542c80ea |
v1beta1Controllers |
success-200-isvc-c85ae |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
e2e-gw1-20542c80ea |
v1beta1Controllers |
success-200-isvc-c85ae |
InferenceServiceReady |
InferenceService [success-200-isvc-c85ae] is Ready | |
e2e-gw1-20542c80ea |
v1beta1Controllers |
error-404-isvc-c85ae |
InferenceServiceReady |
InferenceService [error-404-isvc-c85ae] is Ready | |
| (x5) | e2e-gw1-20542c80ea |
v1beta1Controllers |
error-404-isvc-c85ae |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
e2e-gw1-20542c80ea |
deployment-controller |
sequence-graph-c85ae |
ScalingReplicaSet |
Scaled up replica set sequence-graph-c85ae-78bd9895bb from 0 to 1 | |
e2e-gw1-20542c80ea |
InferenceGraphController |
sequence-graph-c85ae |
InferenceGraphReady |
InferenceGraph [sequence-graph-c85ae] is Ready | |
e2e-gw1-20542c80ea |
replicaset-controller |
sequence-graph-c85ae-78bd9895bb |
SuccessfulCreate |
Created pod: sequence-graph-c85ae-78bd9895bb-qx9sm | |
e2e-gw1-20542c80ea |
kubelet |
sequence-graph-c85ae-78bd9895bb-qx9sm |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:561002e842ac9bbf33141cfcc92eb7b7cc604d16ec35596c7ce81c9e22e080f4" already present on machine and can be accessed by the pod | |
e2e-gw1-20542c80ea |
kubelet |
sequence-graph-c85ae-78bd9895bb-qx9sm |
Created |
Container created | |
e2e-gw1-20542c80ea |
multus |
sequence-graph-c85ae-78bd9895bb-qx9sm |
AddedInterface |
Add eth0 [10.134.0.28/23] from ovn-kubernetes | |
e2e-gw1-20542c80ea |
kubelet |
sequence-graph-c85ae-78bd9895bb-qx9sm |
Started |
Container started | |
| (x3) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
error-404-isvc-c85ae-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
error-404-isvc-c85ae-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
success-200-isvc-c85ae-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
success-200-isvc-c85ae-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
sequence-graph-c85ae |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
sequence-graph-c85ae |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-c85ae-predictor-74dd747476-m87h9 |
Killing |
Stopping container kube-rbac-proxy | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-c85ae-predictor-6db4cb9c4f-gr5x9 |
Killing |
Stopping container kserve-container | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-c85ae-predictor-74dd747476-m87h9 |
Killing |
Stopping container kserve-container | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-c85ae-predictor-6db4cb9c4f-gr5x9 |
Killing |
Stopping container kube-rbac-proxy | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-c85ae-predictor-74dd747476-m87h9 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.40:8643/healthz": dial tcp 10.132.0.40:8643: connect: connection refused | |
| (x8) | e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-c85ae-predictor-74dd747476-m87h9 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.40:8080: connect: connection refused |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-c85ae-predictor-6db4cb9c4f-gr5x9 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.31:8643/healthz": dial tcp 10.133.0.31:8643: connect: connection refused | |
| (x8) | e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-c85ae-predictor-6db4cb9c4f-gr5x9 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.31:8080: connect: connection refused |
e2e-gw1-20542c80ea |
replicaset-controller |
success-200-isvc-f04bb-predictor-5c849f7f88 |
SuccessfulCreate |
Created pod: success-200-isvc-f04bb-predictor-5c849f7f88-rncqq | |
e2e-gw1-20542c80ea |
deployment-controller |
error-404-isvc-f04bb-predictor |
ScalingReplicaSet |
Scaled up replica set error-404-isvc-f04bb-predictor-5cb4948585 from 0 to 1 | |
e2e-gw1-20542c80ea |
v1beta1Controllers |
success-200-isvc-f04bb |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-f04bb": the object has been modified; please apply your changes to the latest version and try again | |
e2e-gw1-20542c80ea |
v1beta1Controllers |
success-200-isvc-f04bb |
UpdateFailed |
Failed to update status for InferenceService "success-200-isvc-f04bb": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "success-200-isvc-f04bb": the object has been modified; please apply your changes to the latest version and try again | |
e2e-gw1-20542c80ea |
replicaset-controller |
error-404-isvc-f04bb-predictor-5cb4948585 |
SuccessfulCreate |
Created pod: error-404-isvc-f04bb-predictor-5cb4948585-xcrsk | |
e2e-gw1-20542c80ea |
deployment-controller |
success-200-isvc-f04bb-predictor |
ScalingReplicaSet |
Scaled up replica set success-200-isvc-f04bb-predictor-5c849f7f88 from 0 to 1 | |
e2e-gw1-20542c80ea |
multus |
success-200-isvc-f04bb-predictor-5c849f7f88-rncqq |
AddedInterface |
Add eth0 [10.132.0.42/23] from ovn-kubernetes | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-f04bb-predictor-5cb4948585-xcrsk |
Created |
Container created | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-f04bb-predictor-5cb4948585-xcrsk |
Pulled |
Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-f04bb-predictor-5c849f7f88-rncqq |
Started |
Container started | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-f04bb-predictor-5cb4948585-xcrsk |
Started |
Container started | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-f04bb-predictor-5c849f7f88-rncqq |
Pulled |
Container image "quay.io/opendatahub/success-200-isvc:odh-pr-2147" already present on machine and can be accessed by the pod | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-f04bb-predictor-5c849f7f88-rncqq |
Created |
Container created | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-f04bb-predictor-5cb4948585-xcrsk |
Created |
Container created | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-f04bb-predictor-5cb4948585-xcrsk |
Pulled |
Container image "quay.io/opendatahub/error-404-isvc:odh-pr-2147" already present on machine and can be accessed by the pod | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-f04bb-predictor-5cb4948585-xcrsk |
Started |
Container started | |
e2e-gw1-20542c80ea |
multus |
error-404-isvc-f04bb-predictor-5cb4948585-xcrsk |
AddedInterface |
Add eth0 [10.133.0.33/23] from ovn-kubernetes | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-f04bb-predictor-5c849f7f88-rncqq |
Pulled |
Container image "quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f" already present on machine and can be accessed by the pod | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-f04bb-predictor-5c849f7f88-rncqq |
Started |
Container started | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-f04bb-predictor-5c849f7f88-rncqq |
Created |
Container created | |
| (x2) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
error-404-isvc-f04bb-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
error-404-isvc-f04bb-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
success-200-isvc-f04bb-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
success-200-isvc-f04bb-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x7) | e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-f04bb-predictor-5cb4948585-xcrsk |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.33:8080: connect: connection refused |
| (x7) | e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-f04bb-predictor-5c849f7f88-rncqq |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.42:8080: connect: connection refused |
e2e-gw1-20542c80ea |
v1beta1Controllers |
success-200-isvc-f04bb |
InferenceServiceReady |
InferenceService [success-200-isvc-f04bb] is Ready | |
| (x5) | e2e-gw1-20542c80ea |
v1beta1Controllers |
success-200-isvc-f04bb |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x5) | e2e-gw1-20542c80ea |
v1beta1Controllers |
error-404-isvc-f04bb |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
e2e-gw1-20542c80ea |
v1beta1Controllers |
error-404-isvc-f04bb |
InferenceServiceReady |
InferenceService [error-404-isvc-f04bb] is Ready | |
e2e-gw1-20542c80ea |
deployment-controller |
switch-graph-f04bb |
ScalingReplicaSet |
Scaled up replica set switch-graph-f04bb-85d9475c98 from 0 to 1 | |
e2e-gw1-20542c80ea |
InferenceGraphController |
switch-graph-f04bb |
InferenceGraphReady |
InferenceGraph [switch-graph-f04bb] is Ready | |
e2e-gw1-20542c80ea |
replicaset-controller |
switch-graph-f04bb-85d9475c98 |
SuccessfulCreate |
Created pod: switch-graph-f04bb-85d9475c98-vrqtr | |
e2e-gw1-20542c80ea |
kubelet |
switch-graph-f04bb-85d9475c98-vrqtr |
Created |
Container created | |
e2e-gw1-20542c80ea |
kubelet |
switch-graph-f04bb-85d9475c98-vrqtr |
Started |
Container started | |
e2e-gw1-20542c80ea |
multus |
switch-graph-f04bb-85d9475c98-vrqtr |
AddedInterface |
Add eth0 [10.134.0.30/23] from ovn-kubernetes | |
e2e-gw1-20542c80ea |
kubelet |
switch-graph-f04bb-85d9475c98-vrqtr |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:561002e842ac9bbf33141cfcc92eb7b7cc604d16ec35596c7ce81c9e22e080f4" already present on machine and can be accessed by the pod | |
| (x3) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
success-200-isvc-f04bb-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
success-200-isvc-f04bb-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
error-404-isvc-f04bb-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
error-404-isvc-f04bb-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
switch-graph-f04bb |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
switch-graph-f04bb |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-f04bb-predictor-5c849f7f88-rncqq |
Killing |
Stopping container kube-rbac-proxy | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-f04bb-predictor-5c849f7f88-rncqq |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.42:8643/healthz": dial tcp 10.132.0.42:8643: connect: connection refused | |
e2e-gw1-20542c80ea |
kubelet |
success-200-isvc-f04bb-predictor-5c849f7f88-rncqq |
Killing |
Stopping container kserve-container | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-f04bb-predictor-5cb4948585-xcrsk |
Killing |
Stopping container kserve-container | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-f04bb-predictor-5cb4948585-xcrsk |
Killing |
Stopping container kube-rbac-proxy | |
e2e-gw1-20542c80ea |
kubelet |
error-404-isvc-f04bb-predictor-5cb4948585-xcrsk |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.33:8643/healthz": dial tcp 10.133.0.33:8643: connect: connection refused | |
| (x4) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
sequence-graph-ff374 |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
sequence-graph-ff374 |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
model-chainer |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
model-chainer |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
switch-graph-f04bb |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | e2e-gw1-20542c80ea |
horizontal-pod-autoscaler |
switch-graph-f04bb |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |