| Time | Namespace | Component | RelatedObject | Reason | Message |
|---|---|---|---|---|---|
kserve-ci-e2e-test |
isvc-canary-stop-v2-predictor-779884cd8f-zpgkf |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf to ip-10-0-142-192.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-cipn-predictor-9774bf55d-nvpfg |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-nvpfg to ip-10-0-131-128.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-097fb-predictor-84cf8468f-x4bcl |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl to ip-10-0-142-192.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-hpa-f37da-578c74c86d-5l9c2 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-hpa-f37da-578c74c86d-5l9c2 to ip-10-0-136-107.ec2.internal | ||
kserve-ci-e2e-test |
isvc-init-fail-96964e-predictor-56bf8fc499-zw626 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626 to ip-10-0-142-192.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5 to ip-10-0-142-192.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm to ip-10-0-142-192.ec2.internal | ||
kserve-ci-e2e-test |
isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff to ip-10-0-142-192.ec2.internal | ||
kserve-ci-e2e-test |
isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk to ip-10-0-142-192.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-predictor-6fcf5b6d4d-fhnw5 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-6fcf5b6d4d-fhnw5 to ip-10-0-131-128.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5 to ip-10-0-142-192.ec2.internal | ||
kserve-ci-e2e-test |
isvc-secondary-d2f65d-predictor-8684c68fdc-zjzbb |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-secondary-d2f65d-predictor-8684c68fdc-zjzbb to ip-10-0-131-128.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-predictor-7879794c64-tvsrl |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl to ip-10-0-142-192.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-849e6-d59cdc6c5-mbfqn |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-849e6-d59cdc6c5-mbfqn to ip-10-0-136-107.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d to ip-10-0-142-192.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-55cddd885b-kmz8m |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m to ip-10-0-142-192.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6 to ip-10-0-142-192.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-v2-predictor-59c98df545-5pwj5 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5 to ip-10-0-142-192.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-55cddd885b-7bjbc |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-7bjbc to ip-10-0-131-128.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-predictor-5cc67bffb8-m4ztq |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq to ip-10-0-142-192.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-v2-predictor-7fb86b4d87-nmf88 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-nmf88 to ip-10-0-131-128.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w to ip-10-0-142-192.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd to ip-10-0-142-192.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh to ip-10-0-142-192.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4 to ip-10-0-142-192.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-hpa-f37da-predictor-5ccd78bf7-875mx |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-f37da-predictor-5ccd78bf7-875mx to ip-10-0-131-128.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-scale-raw-606ec-predictor-5df6959b9f-pfrts |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-scale-raw-606ec-predictor-5df6959b9f-pfrts to ip-10-0-131-128.ec2.internal | ||
kserve-ci-e2e-test |
replicaset-controller |
isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf |
SuccessfulCreate |
Created pod: isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk | |
kserve-ci-e2e-test |
deployment-controller |
isvc-raw-sklearn-batcher-fdd1f-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-fdd1f |
UpdateFailed |
Failed to update status for InferenceService "isvc-raw-sklearn-batcher-fdd1f": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-fdd1f": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-fdd1f |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-raw-sklearn-batcher-fdd1f": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-kmz8m |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-v2-predictor-5f455ffc7c from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-v2-predictor-5f455ffc7c |
SuccessfulCreate |
Created pod: isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh | |
kserve-ci-e2e-test |
multus |
isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk |
AddedInterface |
Add eth0 [10.132.0.18/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:925fa2b887b6743b69728677063d64dcececcf1143340982442b2ec3dc29490e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-55cddd885b |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-55cddd885b-kmz8m | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-predictor-55cddd885b from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-55cddd885b-kmz8m |
AddedInterface |
Add eth0 [10.132.0.19/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-kmz8m |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh |
AddedInterface |
Add eth0 [10.132.0.20/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:925fa2b887b6743b69728677063d64dcececcf1143340982442b2ec3dc29490e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-kmz8m |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-kmz8m |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:925fa2b887b6743b69728677063d64dcececcf1143340982442b2ec3dc29490e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1848" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1848" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-kmz8m |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1848" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-kmz8m |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-kmz8m |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-kmz8m |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1848" in 19.431s (19.431s including waiting). Image size: 1555738027 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1848" in 18.514s (18.514s including waiting). Image size: 1555738027 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-kmz8m |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1848" in 18.485s (18.485s including waiting). Image size: 1555738027 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk |
Pulling |
Pulling image "quay.io/opendatahub/kserve-agent@sha256:bfa995f7c5097489a4c70538ea17c028413cd48f5ccdfc6abfbf73c47a560dc9" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-kmz8m |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.92s (2.92s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-kmz8m |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-kmz8m |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.87s (2.87s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.977s (2.977s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:bfa995f7c5097489a4c70538ea17c028413cd48f5ccdfc6abfbf73c47a560dc9" in 3.418s (3.418s including waiting). Image size: 240196197 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk |
Created |
Created container: agent | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-fdd1f-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-fdd1f-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-kmz8m |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.19:8080: connect: connection refused |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.20:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-fdd1f |
InferenceServiceReady |
InferenceService [isvc-raw-sklearn-batcher-fdd1f] is Ready | |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-fdd1f |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-fdd1f-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-fdd1f-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-predictor-55cddd885b from 1 to 2 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-55cddd885b |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-55cddd885b-7bjbc | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InferenceServiceNotReady |
InferenceService [isvc-canary-rollback] is no longer Ready because of: PredictorReady, Ready, Stopped | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-7bjbc |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:925fa2b887b6743b69728677063d64dcececcf1143340982442b2ec3dc29490e" | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-55cddd885b-7bjbc |
AddedInterface |
Add eth0 [10.134.0.28/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.20:8643/healthz": dial tcp 10.132.0.20:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-7bjbc |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:925fa2b887b6743b69728677063d64dcececcf1143340982442b2ec3dc29490e" in 2.899s (2.899s including waiting). Image size: 302631873 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-7bjbc |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-7bjbc |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-849e6-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-849e6-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-849e6-predictor-65d8f5449 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:925fa2b887b6743b69728677063d64dcececcf1143340982442b2ec3dc29490e" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5 |
AddedInterface |
Add eth0 [10.132.0.21/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-849e6-predictor-65d8f5449 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w |
AddedInterface |
Add eth0 [10.132.0.22/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:925fa2b887b6743b69728677063d64dcececcf1143340982442b2ec3dc29490e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-7bjbc |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1848" | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1848" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.18:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-7bjbc |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1848" in 12.609s (12.609s including waiting). Image size: 1555738027 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-7bjbc |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-7bjbc |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-7bjbc |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-7bjbc |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-7bjbc |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.236s (2.236s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-7bjbc |
Started |
Started container kube-rbac-proxy | |
| (x4) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.18:8643/healthz": dial tcp 10.132.0.18:8643: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 19.525s (19.525s including waiting). Image size: 1307355899 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w |
Created |
Created container: kube-rbac-proxy | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-849e6-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-849e6-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-849e6-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-849e6-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-849e6 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-849e6 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-849e6] is Ready | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-7bjbc |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.28:8080: connect: connection refused |
| (x9) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-849e6 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-849e6 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-849e6] is Ready | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InferenceServiceReady |
InferenceService [isvc-canary-rollback] is Ready |
| (x24) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-849e6-d59cdc6c5-mbfqn |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-raw-849e6-serving-cert" not found | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-849e6 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-849e6] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-849e6 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-849e6-d59cdc6c5 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-849e6-d59cdc6c5 |
SuccessfulCreate |
Created pod: model-chainer-raw-849e6-d59cdc6c5-mbfqn | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-849e6-d59cdc6c5-mbfqn |
Pulling |
Pulling image "quay.io/opendatahub/kserve-router@sha256:9ce7c3058e7df0b4a1b07c83c58a6293353094b90f1440f7751ce51d2cb11c8c" | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-849e6-d59cdc6c5-mbfqn |
AddedInterface |
Add eth0 [10.133.0.29/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-849e6-d59cdc6c5-mbfqn |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:9ce7c3058e7df0b4a1b07c83c58a6293353094b90f1440f7751ce51d2cb11c8c" in 2.302s (2.302s including waiting). Image size: 218259558 bytes. | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-849e6-d59cdc6c5-mbfqn |
Created |
Created container: model-chainer-raw-849e6 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-849e6-d59cdc6c5-mbfqn |
Started |
Started container model-chainer-raw-849e6 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-7bjbc |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-kmz8m |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-kmz8m |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-7bjbc |
Killing |
Stopping container kserve-container | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-849e6-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-55cddd885b-kmz8m |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.19:8643/healthz": dial tcp 10.132.0.19:8643: connect: connection refused | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-849e6-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-849e6-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-849e6-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-6fcf5b6d4d-fhnw5 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-stop-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-hpa-f37da-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-hpa-f37da-predictor-5ccd78bf7 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-hpa-f37da-predictor-5ccd78bf7 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-hpa-f37da-predictor-5ccd78bf7-875mx | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-f37da-predictor-5ccd78bf7-875mx |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-graph-raw-hpa-f37da-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-hpa-f37da-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-849e6-d59cdc6c5-mbfqn |
Killing |
Stopping container model-chainer-raw-849e6 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-predictor-6fcf5b6d4d |
SuccessfulCreate |
Created pod: isvc-canary-stop-predictor-6fcf5b6d4d-fhnw5 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-stop-predictor-6fcf5b6d4d from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-779884cd8f-zpgkf |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-stop-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-stop": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-stop-v2-predictor-779884cd8f from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-v2-predictor-779884cd8f |
SuccessfulCreate |
Created pod: isvc-canary-stop-v2-predictor-779884cd8f-zpgkf | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-graph-raw-hpa-f37da-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-6fcf5b6d4d-fhnw5 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-779884cd8f-zpgkf |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-predictor-6fcf5b6d4d-fhnw5 |
AddedInterface |
Add eth0 [10.134.0.29/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-f37da-predictor-5ccd78bf7-875mx |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-6fcf5b6d4d-fhnw5 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-hpa-f37da-predictor-5ccd78bf7-875mx |
AddedInterface |
Add eth0 [10.134.0.30/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.22:8643/healthz": dial tcp 10.132.0.22:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-779884cd8f-zpgkf |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-6fcf5b6d4d-fhnw5 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:925fa2b887b6743b69728677063d64dcececcf1143340982442b2ec3dc29490e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-779884cd8f-zpgkf |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:925fa2b887b6743b69728677063d64dcececcf1143340982442b2ec3dc29490e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:925fa2b887b6743b69728677063d64dcececcf1143340982442b2ec3dc29490e" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-v2-predictor-779884cd8f-zpgkf |
AddedInterface |
Add eth0 [10.132.0.23/23] from ovn-kubernetes | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.22:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-f37da-predictor-5ccd78bf7-875mx |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:925fa2b887b6743b69728677063d64dcececcf1143340982442b2ec3dc29490e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-f37da-predictor-5ccd78bf7-875mx |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6 |
AddedInterface |
Add eth0 [10.132.0.24/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.21:8643/healthz": dial tcp 10.132.0.21:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.21:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-779884cd8f-zpgkf |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6 |
Pulled |
Container image "kserve/xgbserver:latest" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-779884cd8f-zpgkf |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-779884cd8f-zpgkf |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1848" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-779884cd8f-zpgkf |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-779884cd8f-zpgkf |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-779884cd8f-zpgkf |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-f37da-predictor-5ccd78bf7-875mx |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-6fcf5b6d4d-fhnw5 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-6fcf5b6d4d-fhnw5 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-6fcf5b6d4d-fhnw5 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-6fcf5b6d4d-fhnw5 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-6fcf5b6d4d-fhnw5 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-f37da-predictor-5ccd78bf7-875mx |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-f37da-predictor-5ccd78bf7-875mx |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-f37da-predictor-5ccd78bf7-875mx |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-f37da-predictor-5ccd78bf7-875mx |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-f37da-predictor-5ccd78bf7-875mx |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1848" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-6fcf5b6d4d-fhnw5 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1848" already present on machine | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-849e6-d59cdc6c5-mbfqn |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-f37da-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-f37da-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-f37da-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-f37da-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-f37da |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-hpa-f37da] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-f37da |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-779884cd8f-zpgkf |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.23:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-f37da |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-hpa-f37da] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-f37da |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InferenceServiceReady |
InferenceService [isvc-canary-stop] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-779884cd8f-zpgkf |
Killing |
Stopping container kserve-container | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-f37da-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-6fcf5b6d4d-fhnw5 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-6fcf5b6d4d-fhnw5 |
Killing |
Stopping container kserve-container | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-f37da-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-779884cd8f-zpgkf |
Killing |
Stopping container kube-rbac-proxy | |
| (x16) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InferenceServiceNotReady |
InferenceService [isvc-canary-stop] is no longer Ready because of: PredictorReady, Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-6fcf5b6d4d-fhnw5 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.29:8643/healthz": dial tcp 10.134.0.29:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-779884cd8f-zpgkf |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.23:8643/healthz": dial tcp 10.132.0.23:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-6fcf5b6d4d-fhnw5 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.29:8080: connect: connection refused |
kserve-ci-e2e-test |
deployment-controller |
isvc-primary-d2f65d-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-primary-d2f65d-predictor-74b464cbd4 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-primary-d2f65d-predictor-74b464cbd4 |
SuccessfulCreate |
Created pod: isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-primary-d2f65d-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-hpa-f37da |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-hpa-f37da-578c74c86d from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-hpa-f37da-578c74c86d |
SuccessfulCreate |
Created pod: model-chainer-raw-hpa-f37da-578c74c86d-5l9c2 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-d2f65d |
UpdateFailed |
Failed to update status for InferenceService "isvc-primary-d2f65d": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-d2f65d": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-d2f65d |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-d2f65d": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-f37da |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-hpa-f37da] is Ready | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-f37da-578c74c86d-5l9c2 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-raw-hpa-f37da-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-f37da-578c74c86d-5l9c2 |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:9ce7c3058e7df0b4a1b07c83c58a6293353094b90f1440f7751ce51d2cb11c8c" already present on machine | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-f37da-578c74c86d-5l9c2 |
Created |
Created container: model-chainer-raw-hpa-f37da | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-f37da-578c74c86d-5l9c2 |
Started |
Started container model-chainer-raw-hpa-f37da | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-hpa-f37da-578c74c86d-5l9c2 |
AddedInterface |
Add eth0 [10.133.0.30/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff |
AddedInterface |
Add eth0 [10.132.0.25/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:925fa2b887b6743b69728677063d64dcececcf1143340982442b2ec3dc29490e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff |
Created |
Created container: storage-initializer | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-f37da-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-f37da-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1848" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-f37da-predictor-5ccd78bf7-875mx |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-f37da-predictor-5ccd78bf7-875mx |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-f99f6-predictor-7855974cd8 |
SuccessfulCreate |
Created pod: message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d |
AddedInterface |
Add eth0 [10.132.0.26/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-f37da-578c74c86d-5l9c2 |
Killing |
Stopping container model-chainer-raw-hpa-f37da | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-f99f6-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-f99f6-predictor-7855974cd8 from 0 to 1 | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-f37da-predictor-5ccd78bf7-875mx |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.30:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-f99f6 |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-f99f6": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-f99f6": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d |
Pulling |
Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d |
Pulled |
Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.172s (1.172s including waiting). Image size: 14813193 bytes. | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.24:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.24:8643/healthz": dial tcp 10.132.0.24:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-f37da-predictor-5ccd78bf7-875mx |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.30:8643/healthz": dial tcp 10.134.0.30:8643: connect: connection refused | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-f99f6 |
InferenceServiceReady |
InferenceService [message-dumper-raw-f99f6] is Ready | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-f99f6-predictor-765fd8c79b |
SuccessfulCreate |
Created pod: isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm | |
| (x7) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-f99f6 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-f99f6-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-f99f6-predictor-765fd8c79b from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-f99f6 |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-f99f6": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-f99f6": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:925fa2b887b6743b69728677063d64dcececcf1143340982442b2ec3dc29490e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm |
AddedInterface |
Add eth0 [10.132.0.27/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:bfa995f7c5097489a4c70538ea17c028413cd48f5ccdfc6abfbf73c47a560dc9" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm |
Started |
Started container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1848" already present on machine | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-f99f6-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-f99f6-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-f37da-578c74c86d-5l9c2 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-f99f6-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-f99f6-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-d2f65d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-d2f65d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.25:8080: connect: connection refused |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-d2f65d |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-d2f65d |
InferenceServiceReady |
InferenceService [isvc-primary-d2f65d] is Ready | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-f99f6 |
InferenceServiceReady |
InferenceService [isvc-logger-raw-f99f6] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-f99f6 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
replicaset-controller |
isvc-secondary-d2f65d-predictor-8684c68fdc |
SuccessfulCreate |
Created pod: isvc-secondary-d2f65d-predictor-8684c68fdc-zjzbb | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-d2f65d |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-d2f65d": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-d2f65d |
UpdateFailed |
Failed to update status for InferenceService "isvc-secondary-d2f65d": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-d2f65d": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-secondary-d2f65d-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-secondary-d2f65d-predictor-8684c68fdc from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-d2f65d-predictor-8684c68fdc-zjzbb |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-secondary-d2f65d-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
multus |
isvc-secondary-d2f65d-predictor-8684c68fdc-zjzbb |
AddedInterface |
Add eth0 [10.134.0.31/23] from ovn-kubernetes | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-f99f6-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-f99f6-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-d2f65d-predictor-8684c68fdc-zjzbb |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-d2f65d-predictor-8684c68fdc-zjzbb |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:925fa2b887b6743b69728677063d64dcececcf1143340982442b2ec3dc29490e" already present on machine |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-d2f65d-predictor-8684c68fdc-zjzbb |
Created |
Created container: storage-initializer |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-scale-raw-606ec-predictor-5df6959b9f |
SuccessfulCreate |
Created pod: isvc-sklearn-scale-raw-606ec-predictor-5df6959b9f-pfrts | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-scale-raw-606ec-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-scale-raw-606ec-predictor-5df6959b9f from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm |
Killing |
Stopping container agent | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-606ec-predictor-5df6959b9f-pfrts |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-scale-raw-606ec-predictor-serving-cert" not found |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-606ec-predictor-5df6959b9f-pfrts |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:925fa2b887b6743b69728677063d64dcececcf1143340982442b2ec3dc29490e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-606ec-predictor-5df6959b9f-pfrts |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-scale-raw-606ec-predictor-5df6959b9f-pfrts |
AddedInterface |
Add eth0 [10.134.0.32/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-606ec-predictor-5df6959b9f-pfrts |
Started |
Started container storage-initializer | |
| (x13) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-d2f65d |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-d2f65d-predictor-8684c68fdc-zjzbb |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-secondary-d2f65d-predictor-8684c68fdc-zjzbb_kserve-ci-e2e-test(c353ea9c-ae87-47ba-adf6-a424ebc56c25) | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-d2f65d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-d2f65d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-d2f65d-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-d2f65d-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-606ec-predictor-5df6959b9f-pfrts |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-606ec-predictor-5df6959b9f-pfrts |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-606ec-predictor-5df6959b9f-pfrts |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-606ec-predictor-5df6959b9f-pfrts |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1848" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-606ec-predictor-5df6959b9f-pfrts |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-606ec-predictor-5df6959b9f-pfrts |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff |
Killing |
Stopping container kserve-container | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.27:8643/healthz": dial tcp 10.132.0.27:8643: connect: connection refused |
| (x12) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.27:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.25:8643/healthz": dial tcp 10.132.0.25:8643: connect: connection refused | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-96964e |
UpdateFailed |
Failed to update status for InferenceService "isvc-init-fail-96964e": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-96964e": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-96964e |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-96964e": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
replicaset-controller |
isvc-init-fail-96964e-predictor-56bf8fc499 |
SuccessfulCreate |
Created pod: isvc-init-fail-96964e-predictor-56bf8fc499-zw626 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-init-fail-96964e-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-init-fail-96964e-predictor-56bf8fc499 from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-init-fail-96964e-predictor-56bf8fc499-zw626 |
AddedInterface |
Add eth0 [10.132.0.28/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-96964e-predictor-56bf8fc499-zw626 |
Started |
Started container storage-initializer |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-96964e |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-96964e-predictor-56bf8fc499-zw626 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:925fa2b887b6743b69728677063d64dcececcf1143340982442b2ec3dc29490e" already present on machine |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-96964e-predictor-56bf8fc499-zw626 |
Created |
Created container: storage-initializer |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-96964e-predictor-56bf8fc499-zw626 |
FailedToRetrieveImagePullSecret |
Unable to retrieve some image pull secrets (fail-s3-sa-96964e-dockercfg-xzh6h); attempting to pull the image may not succeed. | |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-96964e-predictor-56bf8fc499-zw626 |
Killing |
Stopping container storage-initializer | |
| (x5) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-96964e-predictor-56bf8fc499-zw626 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-init-fail-96964e-predictor-serving-cert" not found |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-097fb-predictor-84cf8468f |
SuccessfulCreate |
Created pod: raw-sklearn-097fb-predictor-84cf8468f-x4bcl | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-097fb-predictor-84cf8468f-x4bcl |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-097fb-predictor-84cf8468f-x4bcl |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-097fb-predictor-84cf8468f-x4bcl |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:925fa2b887b6743b69728677063d64dcececcf1143340982442b2ec3dc29490e" already present on machine | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-097fb |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-097fb": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-097fb": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
multus |
raw-sklearn-097fb-predictor-84cf8468f-x4bcl |
AddedInterface |
Add eth0 [10.132.0.29/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-097fb |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-097fb": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-097fb-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-097fb-predictor-84cf8468f from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-097fb-predictor-84cf8468f-x4bcl |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-097fb-predictor-84cf8468f-x4bcl |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-097fb-predictor-84cf8468f-x4bcl |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1848" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-097fb-predictor-84cf8468f-x4bcl |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-097fb-predictor-84cf8468f-x4bcl |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-097fb-predictor-84cf8468f-x4bcl |
Started |
Started container kserve-container | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-606ec-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-606ec-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-097fb-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-097fb-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-097fb |
InferenceServiceReady |
InferenceService [raw-sklearn-097fb] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-097fb |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-097fb-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-097fb-predictor-84cf8468f-x4bcl |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-097fb-predictor-84cf8468f-x4bcl |
Killing |
Stopping container kube-rbac-proxy | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-097fb-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-097fb-predictor-84cf8468f-x4bcl |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.29:8643/healthz": dial tcp 10.132.0.29:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-097fb-predictor-84cf8468f-x4bcl |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.29:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:925fa2b887b6743b69728677063d64dcececcf1143340982442b2ec3dc29490e" already present on machine | |
kserve-ci-e2e-test |
multus |
raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4 |
AddedInterface |
Add eth0 [10.132.0.30/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-runtime-992fe-predictor-6d7b9fcc49 |
SuccessfulCreate |
Created pod: raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4 | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-runtime-992fe-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-runtime-992fe-predictor-6d7b9fcc49 from 0 to 1 | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-992fe |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-runtime-992fe": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-992fe": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-992fe |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-992fe": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1848" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-992fe-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-992fe-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x18) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-606ec-predictor-5df6959b9f-pfrts |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.32:8080: connect: connection refused |
| (x14) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-606ec |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-606ec |
InferenceServiceReady |
InferenceService [isvc-sklearn-scale-raw-606ec] is Ready | |
| (x9) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-606ec-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x9) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-606ec-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-606ec-predictor-5df6959b9f-pfrts |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-606ec-predictor-5df6959b9f-pfrts |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-create-v2-predictor-59c98df545 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-v2-predictor-59c98df545 |
SuccessfulCreate |
Created pod: isvc-canary-create-v2-predictor-59c98df545-5pwj5 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-create": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-predictor-7879794c64-tvsrl |
AddedInterface |
Add eth0 [10.132.0.31/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7879794c64-tvsrl |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:925fa2b887b6743b69728677063d64dcececcf1143340982442b2ec3dc29490e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7879794c64-tvsrl |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-predictor-7879794c64 |
SuccessfulCreate |
Created pod: isvc-canary-create-predictor-7879794c64-tvsrl | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7879794c64-tvsrl |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-59c98df545-5pwj5 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-59c98df545-5pwj5 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-59c98df545-5pwj5 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:925fa2b887b6743b69728677063d64dcececcf1143340982442b2ec3dc29490e" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-v2-predictor-59c98df545-5pwj5 |
AddedInterface |
Add eth0 [10.132.0.32/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-create-predictor-7879794c64 from 0 to 1 | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-606ec-predictor-5df6959b9f-pfrts |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.32:8643/healthz": dial tcp 10.134.0.32:8643: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7879794c64-tvsrl |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-59c98df545-5pwj5 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1848" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7879794c64-tvsrl |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-59c98df545-5pwj5 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-59c98df545-5pwj5 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-59c98df545-5pwj5 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-59c98df545-5pwj5 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-59c98df545-5pwj5 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7879794c64-tvsrl |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7879794c64-tvsrl |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1848" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7879794c64-tvsrl |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7879794c64-tvsrl |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-992fe |
InferenceServiceReady |
InferenceService [raw-sklearn-runtime-992fe] is Ready | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-992fe |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4 |
Killing |
Stopping container kserve-container | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-992fe-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-992fe-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x11) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.30:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.30:8643/healthz": dial tcp 10.132.0.30:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7879794c64-tvsrl |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.31:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
InferenceServiceReady |
InferenceService [isvc-canary-create] is Ready | |
| (x14) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-59c98df545-5pwj5 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-59c98df545-5pwj5 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7879794c64-tvsrl |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7879794c64-tvsrl |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-59c98df545-5pwj5 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.32:8643/healthz": dial tcp 10.132.0.32:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-59c98df545-5pwj5 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.32:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-7879794c64-tvsrl |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.31:8643/healthz": dial tcp 10.132.0.31:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5cc67bffb8-m4ztq |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-promote-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-v2-predictor-7fb86b4d87 |
SuccessfulCreate |
Created pod: isvc-canary-promote-v2-predictor-7fb86b4d87-nmf88 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-nmf88 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-promote-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-v2-predictor-7fb86b4d87 from 0 to 1 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-predictor-5cc67bffb8 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-predictor-5cc67bffb8 |
SuccessfulCreate |
Created pod: isvc-canary-promote-predictor-5cc67bffb8-m4ztq | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5cc67bffb8-m4ztq |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-predictor-5cc67bffb8-m4ztq |
AddedInterface |
Add eth0 [10.132.0.33/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-nmf88 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-nmf88 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-nmf88 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:925fa2b887b6743b69728677063d64dcececcf1143340982442b2ec3dc29490e" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-v2-predictor-7fb86b4d87-nmf88 |
AddedInterface |
Add eth0 [10.134.0.33/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5cc67bffb8-m4ztq |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5cc67bffb8-m4ztq |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:925fa2b887b6743b69728677063d64dcececcf1143340982442b2ec3dc29490e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5cc67bffb8-m4ztq |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5cc67bffb8-m4ztq |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5cc67bffb8-m4ztq |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5cc67bffb8-m4ztq |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5cc67bffb8-m4ztq |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1848" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5cc67bffb8-m4ztq |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-nmf88 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1848" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-nmf88 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-nmf88 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-nmf88 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-nmf88 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-nmf88 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-v2-predictor-7fb86b4d87 from 1 to 2 | |
| (x20) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-promote": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InferenceServiceNotReady |
InferenceService [isvc-canary-promote] is no longer Ready because of: PredictorReady, Ready, Stopped | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-v2-predictor-7fb86b4d87 |
SuccessfulCreate |
Created pod: isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5cc67bffb8-m4ztq |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5cc67bffb8-m4ztq |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:925fa2b887b6743b69728677063d64dcececcf1143340982442b2ec3dc29490e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd |
AddedInterface |
Add eth0 [10.132.0.34/23] from ovn-kubernetes | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5cc67bffb8-m4ztq |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.33:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-5cc67bffb8-m4ztq |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.33:8643/healthz": dial tcp 10.132.0.33:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1848" already present on machine | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InferenceServiceReady |
InferenceService [isvc-canary-promote] is Ready |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-nmf88 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-nmf88 |
Killing |
Stopping container kube-rbac-proxy | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.34:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.34:8643/healthz": dial tcp 10.132.0.34:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-nmf88 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.33:8643/healthz": dial tcp 10.134.0.33:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-7fb86b4d87-nmf88 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.33:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-cipn |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-cipn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-cipn": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-cipn-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-cipn-predictor-9774bf55d from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-nvpfg |
Pulling |
Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-cipn-predictor-9774bf55d-nvpfg |
AddedInterface |
Add eth0 [10.134.0.34/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-cipn-predictor-9774bf55d |
SuccessfulCreate |
Created pod: message-dumper-raw-cipn-predictor-9774bf55d-nvpfg | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-nvpfg |
Pulled |
Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.171s (1.171s including waiting). Image size: 14813193 bytes. | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-nvpfg |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-nvpfg |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-nvpfg |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-nvpfg |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-nvpfg |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-cipn |
InferenceServiceReady |
InferenceService [message-dumper-raw-cipn] is Ready | |
| (x7) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-cipn |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-cipn-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-cipn-predictor-7d4b6c5f84 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-logger-raw-cipn-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-cipn |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-cipn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-cipn": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-cipn-predictor-7d4b6c5f84 |
SuccessfulCreate |
Created pod: isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5 | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:925fa2b887b6743b69728677063d64dcececcf1143340982442b2ec3dc29490e" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5 |
AddedInterface |
Add eth0 [10.132.0.35/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5 |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:bfa995f7c5097489a4c70538ea17c028413cd48f5ccdfc6abfbf73c47a560dc9" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1848" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5 |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5 |
Started |
Started container agent | |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-cipn-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-cipn-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-cipn-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-cipn-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-cipn |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-cipn |
InferenceServiceReady |
InferenceService [isvc-logger-raw-cipn] is Ready | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-cipn-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-cipn-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5 |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-nvpfg |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-nvpfg |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5 |
Killing |
Stopping container kube-rbac-proxy | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5 |
Unhealthy |
Readiness probe failed: dial tcp 10.132.0.35:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5 |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x4) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5 |
Unhealthy |
Readiness probe failed: Get "https://10.132.0.35:8643/healthz": dial tcp 10.132.0.35:8643: connect: connection refused |