| Time | Namespace | Component | RelatedObject | Reason | Message |
|---|---|---|---|---|---|
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5 to ip-10-0-132-85.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-cipn-predictor-9774bf55d-8dskw |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw to ip-10-0-132-85.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-predictor-8bc5b857-gkln2 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-gkln2 to ip-10-0-130-106.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm to ip-10-0-132-85.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-predictor-67d6bcd4b5-kmt4g |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-kmt4g to ip-10-0-130-106.ec2.internal | ||
kserve-ci-e2e-test |
isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q to ip-10-0-132-85.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-696c5cc6cc-qrqx8 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-qrqx8 to ip-10-0-130-106.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh to ip-10-0-132-85.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-696c5cc6cc-rwk9c |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c to ip-10-0-132-85.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-predictor-67d6bcd4b5-bwhcb |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb to ip-10-0-132-85.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-v2-predictor-65744c97b6-v7xcp |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-v7xcp to ip-10-0-130-106.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-hpa-48552-86d85b86d-lrnss |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-hpa-48552-86d85b86d-lrnss to ip-10-0-130-106.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-predictor-696c5cc6cc-gbf7p |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p to ip-10-0-132-85.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-predictor-8bc5b857-9fh8b |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b to ip-10-0-132-85.ec2.internal | ||
kserve-ci-e2e-test |
isvc-init-fail-d41ccc-predictor-7cc64b44bd-2k4qk |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-init-fail-d41ccc-predictor-7cc64b44bd-2k4qk to ip-10-0-130-106.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-predictor-8cb968449-hkr9h |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h to ip-10-0-132-85.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-stop-v2-predictor-775495c6b4-np76t |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t to ip-10-0-132-85.ec2.internal | ||
kserve-ci-e2e-test |
isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk to ip-10-0-132-85.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-predictor-8cb968449-cxzjv |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-cxzjv to ip-10-0-130-106.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2 to ip-10-0-132-85.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz to ip-10-0-132-85.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g to ip-10-0-132-85.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-promote-v2-predictor-65744c97b6-2gm85 |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85 to ip-10-0-132-85.ec2.internal | ||
kserve-ci-e2e-test |
isvc-primary-42374c-predictor-798b4f4b89-bb7vt |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt to ip-10-0-132-85.ec2.internal | ||
kserve-ci-e2e-test |
isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm to ip-10-0-132-85.ec2.internal | ||
kserve-ci-e2e-test |
isvc-secondary-42374c-predictor-86544c997d-ch6dv |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-secondary-42374c-predictor-86544c997d-ch6dv to ip-10-0-130-106.ec2.internal | ||
kserve-ci-e2e-test |
isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r to ip-10-0-132-85.ec2.internal | ||
kserve-ci-e2e-test |
isvc-canary-create-v2-predictor-7768896644-v24qh |
Scheduled |
Successfully assigned kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh to ip-10-0-132-85.ec2.internal | ||
kserve-ci-e2e-test |
message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn |
Scheduled |
Successfully assigned kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn to ip-10-0-132-85.ec2.internal | ||
kserve-ci-e2e-test |
model-chainer-raw-48e77-77c5845d5-h5tqd |
Scheduled |
Successfully assigned kserve-ci-e2e-test/model-chainer-raw-48e77-77c5845d5-h5tqd to ip-10-0-130-106.ec2.internal | ||
kserve-ci-e2e-test |
raw-sklearn-f0e12-predictor-8656cc4776-v9r9w |
Scheduled |
Successfully assigned kserve-ci-e2e-test/raw-sklearn-f0e12-predictor-8656cc4776-v9r9w to ip-10-0-130-106.ec2.internal | ||
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-v2-predictor-7ffd87447b from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-v2-predictor-7ffd87447b |
SuccessfulCreate |
Created pod: isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b |
SuccessfulCreate |
Created pod: isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-696c5cc6cc-qrqx8 |
AddedInterface |
Add eth0 [10.133.0.29/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-696c5cc6cc |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-696c5cc6cc-gbf7p | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-qrqx8 |
Pulling |
Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-gbf7p |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-696c5cc6cc |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-696c5cc6cc-qrqx8 | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-696c5cc6cc-gbf7p |
AddedInterface |
Add eth0 [10.134.0.20/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
isvc-raw-sklearn-batcher-5ef83-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-raw-sklearn-batcher-5ef83-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-predictor-696c5cc6cc from 0 to 2 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-gbf7p |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-gbf7p |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm |
AddedInterface |
Add eth0 [10.134.0.21/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz |
AddedInterface |
Add eth0 [10.134.0.19/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-qrqx8 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-qrqx8 |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" in 3.261s (3.261s including waiting). Image size: 306673092 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-qrqx8 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-gbf7p |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-qrqx8 |
Pulling |
Pulling image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-qrqx8 |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-qrqx8 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-qrqx8 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-qrqx8 |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" in 13.061s (13.061s including waiting). Image size: 1580728223 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-qrqx8 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-qrqx8 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-qrqx8 |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.152s (2.152s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-gbf7p |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-gbf7p |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-gbf7p |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" in 21.768s (21.768s including waiting). Image size: 1580728223 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-gbf7p |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" in 21.209s (21.209s including waiting). Image size: 1580728223 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm |
Pulled |
Successfully pulled image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" in 20.286s (20.286s including waiting). Image size: 1580728223 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm |
Pulling |
Pulling image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-gbf7p |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm |
Pulling |
Pulling image "quay.io/opendatahub/kserve-agent@sha256:6b08d0c6feafdc947d16756a5e87ed5c77db04b24369e3640cb13be40670b327" | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-gbf7p |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 2.623s (2.623s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 3.002s (3.002s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-gbf7p |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm |
Pulled |
Successfully pulled image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" in 3.021s (3.021s including waiting). Image size: 211946088 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-agent@sha256:6b08d0c6feafdc947d16756a5e87ed5c77db04b24369e3640cb13be40670b327" in 2.528s (2.528s including waiting). Image size: 240302180 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm |
Started |
Started container agent | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-5ef83-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-5ef83-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.19:8080: connect: connection refused |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-gbf7p |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.20:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-5ef83 |
InferenceServiceReady |
InferenceService [isvc-raw-sklearn-batcher-5ef83] is Ready | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-raw-sklearn-batcher-5ef83 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-5ef83-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-raw-sklearn-batcher-5ef83-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled down replica set isvc-canary-rollback-predictor-696c5cc6cc from 2 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-gbf7p |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-gbf7p |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-696c5cc6cc |
SuccessfulDelete |
Deleted pod: isvc-canary-rollback-predictor-696c5cc6cc-gbf7p | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-rollback": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-rollback": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-rollback-predictor-696c5cc6cc |
SuccessfulCreate |
Created pod: isvc-canary-rollback-predictor-696c5cc6cc-rwk9c | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz |
Killing |
Stopping container kserve-container | |
| (x21) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-rollback-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-rollback-predictor-696c5cc6cc from 1 to 2 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-rwk9c |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-rwk9c |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-rwk9c |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-rollback-predictor-696c5cc6cc-rwk9c |
AddedInterface |
Add eth0 [10.134.0.22/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.19:8643/healthz": dial tcp 10.134.0.19:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-rwk9c |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-rwk9c |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-rwk9c |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-rwk9c |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-rwk9c |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-rwk9c |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk |
AddedInterface |
Add eth0 [10.134.0.24/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-48e77-predictor-5477d668d5 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-48e77-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-48e77-predictor-5477d668d5 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-48e77 |
UpdateFailed |
Failed to update status for InferenceService "isvc-sklearn-graph-raw-48e77": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-sklearn-graph-raw-48e77": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r |
AddedInterface |
Add eth0 [10.134.0.23/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-48e77-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-48e77-predictor-759d457548 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-48e77-predictor-759d457548 |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk |
Pulling |
Pulling image "kserve/xgbserver:latest" | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.21:8643/healthz": dial tcp 10.134.0.21:8643: connect: connection refused |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.21:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk |
Pulled |
Successfully pulled image "kserve/xgbserver:latest" in 20.388s (20.388s including waiting). Image size: 1435742979 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk |
Started |
Started container kube-rbac-proxy | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-48e77-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-48e77-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-48e77-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-48e77-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-rollback |
InferenceServiceReady |
InferenceService [isvc-canary-rollback] is Ready |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-48e77 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-48e77 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-48e77] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-qrqx8 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-qrqx8 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-rwk9c |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-rwk9c |
Killing |
Stopping container kserve-container | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-qrqx8 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.29:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-qrqx8 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.29:8643/healthz": dial tcp 10.133.0.29:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-rwk9c |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.22:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-rollback-predictor-696c5cc6cc-rwk9c |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.22:8643/healthz": dial tcp 10.134.0.22:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-kmt4g |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-kmt4g |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-775495c6b4-np76t |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-775495c6b4-np76t |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-775495c6b4-np76t |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-v2-predictor-775495c6b4-np76t |
AddedInterface |
Add eth0 [10.134.0.25/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-stop-predictor-67d6bcd4b5 from 0 to 2 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-predictor-67d6bcd4b5 |
SuccessfulCreate |
Created pod: isvc-canary-stop-predictor-67d6bcd4b5-bwhcb | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-bwhcb |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-bwhcb |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-predictor-67d6bcd4b5 |
SuccessfulCreate |
Created pod: isvc-canary-stop-predictor-67d6bcd4b5-kmt4g | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-v2-predictor-775495c6b4 |
SuccessfulCreate |
Created pod: isvc-canary-stop-v2-predictor-775495c6b4-np76t | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-stop-v2-predictor-775495c6b4 from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-predictor-67d6bcd4b5-bwhcb |
AddedInterface |
Add eth0 [10.134.0.26/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-kmt4g |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-stop-predictor-67d6bcd4b5-kmt4g |
AddedInterface |
Add eth0 [10.133.0.30/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-bwhcb |
Started |
Started container storage-initializer | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-48e77 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-48e77 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-48e77] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-kmt4g |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-bwhcb |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-bwhcb |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-775495c6b4-np76t |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-775495c6b4-np76t |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-bwhcb |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-bwhcb |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-bwhcb |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-775495c6b4-np76t |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-775495c6b4-np76t |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-775495c6b4-np76t |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-bwhcb |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-775495c6b4-np76t |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-kmt4g |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-kmt4g |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-kmt4g |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-kmt4g |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-kmt4g |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-48e77 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-48e77] is Ready | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-48e77-77c5845d5-h5tqd |
Pulling |
Pulling image "quay.io/opendatahub/kserve-router@sha256:dae09a3aa9a23aca96ea2af0400a27c12957dcc91259106b40cbd871a02ad5c3" | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-48e77-77c5845d5-h5tqd |
AddedInterface |
Add eth0 [10.133.0.31/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-48e77 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-48e77-77c5845d5 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-48e77-77c5845d5 |
SuccessfulCreate |
Created pod: model-chainer-raw-48e77-77c5845d5-h5tqd | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-48e77-77c5845d5-h5tqd |
Created |
Created container: model-chainer-raw-48e77 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-48e77-77c5845d5-h5tqd |
Pulled |
Successfully pulled image "quay.io/opendatahub/kserve-router@sha256:dae09a3aa9a23aca96ea2af0400a27c12957dcc91259106b40cbd871a02ad5c3" in 2.219s (2.219s including waiting). Image size: 211751527 bytes. | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-48e77-77c5845d5-h5tqd |
Started |
Started container model-chainer-raw-48e77 | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-48e77-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-48e77-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-48e77-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-48e77-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-48e77-77c5845d5-h5tqd |
Killing |
Stopping container model-chainer-raw-48e77 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-graph-raw-hpa-48552-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-xgboost-graph-raw-hpa-48552-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d |
SuccessfulCreate |
Created pod: isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-xgboost-graph-raw-hpa-48552-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997 |
SuccessfulCreate |
Created pod: isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-graph-raw-hpa-48552-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5 |
AddedInterface |
Add eth0 [10.134.0.27/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
multus |
isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh |
AddedInterface |
Add eth0 [10.134.0.28/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.23:8643/healthz": dial tcp 10.134.0.23:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.23:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.24:8643/healthz": dial tcp 10.134.0.24:8643: connect: connection refused | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.24:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh |
Pulled |
Container image "kserve/xgbserver:latest" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh |
Created |
Created container: kube-rbac-proxy | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-48e77-77c5845d5-h5tqd |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-48552-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-48552-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-48552-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-48552-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-kmt4g |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.30:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-bwhcb |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.26:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-kmt4g |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.30:8643/healthz": dial tcp 10.133.0.30:8643: connect: connection refused | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-stop-predictor |
ScalingReplicaSet |
Scaled down replica set isvc-canary-stop-predictor-67d6bcd4b5 from 2 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-kmt4g |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-stop-predictor-67d6bcd4b5 |
SuccessfulDelete |
Deleted pod: isvc-canary-stop-predictor-67d6bcd4b5-kmt4g | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-stop": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-stop": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-kmt4g |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
InferenceServiceReady |
InferenceService [isvc-canary-stop] is Ready | |
| (x21) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-stop |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-bwhcb |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-bwhcb |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-775495c6b4-np76t |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-775495c6b4-np76t |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-775495c6b4-np76t |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.25:8643/healthz": dial tcp 10.134.0.25:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-v2-predictor-775495c6b4-np76t |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.25:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-stop-predictor-67d6bcd4b5-bwhcb |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.26:8643/healthz": dial tcp 10.134.0.26:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-48552 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-xgboost-graph-raw-hpa-48552 |
InferenceServiceReady |
InferenceService [isvc-xgboost-graph-raw-hpa-48552] is Ready | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-42374c |
UpdateFailed |
Failed to update status for InferenceService "isvc-primary-42374c": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-42374c": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-primary-42374c-predictor-798b4f4b89 |
SuccessfulCreate |
Created pod: isvc-primary-42374c-predictor-798b4f4b89-bb7vt | |
kserve-ci-e2e-test |
deployment-controller |
isvc-primary-42374c-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-primary-42374c-predictor-798b4f4b89 from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-42374c |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-primary-42374c": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-42374c-predictor-798b4f4b89-bb7vt |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-primary-42374c-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
multus |
isvc-primary-42374c-predictor-798b4f4b89-bb7vt |
AddedInterface |
Add eth0 [10.134.0.29/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-42374c-predictor-798b4f4b89-bb7vt |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-42374c-predictor-798b4f4b89-bb7vt |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-42374c-predictor-798b4f4b89-bb7vt |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine | |
| (x10) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-48552 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-graph-raw-hpa-48552 |
InferenceServiceReady |
InferenceService [isvc-sklearn-graph-raw-hpa-48552] is Ready | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-42374c-predictor-798b4f4b89-bb7vt |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-42374c-predictor-798b4f4b89-bb7vt |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-42374c-predictor-798b4f4b89-bb7vt |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-42374c-predictor-798b4f4b89-bb7vt |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-42374c-predictor-798b4f4b89-bb7vt |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-42374c-predictor-798b4f4b89-bb7vt |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" already present on machine | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-48552-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-xgboost-graph-raw-hpa-48552-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
replicaset-controller |
model-chainer-raw-hpa-48552-86d85b86d |
SuccessfulCreate |
Created pod: model-chainer-raw-hpa-48552-86d85b86d-lrnss | |
kserve-ci-e2e-test |
InferenceGraphController |
model-chainer-raw-hpa-48552 |
InferenceGraphReady |
InferenceGraph [model-chainer-raw-hpa-48552] is Ready | |
kserve-ci-e2e-test |
deployment-controller |
model-chainer-raw-hpa-48552 |
ScalingReplicaSet |
Scaled up replica set model-chainer-raw-hpa-48552-86d85b86d from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-48552-86d85b86d-lrnss |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "model-chainer-raw-hpa-48552-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-48552-86d85b86d-lrnss |
Started |
Started container model-chainer-raw-hpa-48552 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-48552-86d85b86d-lrnss |
Created |
Created container: model-chainer-raw-hpa-48552 | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-48552-86d85b86d-lrnss |
Pulled |
Container image "quay.io/opendatahub/kserve-router@sha256:dae09a3aa9a23aca96ea2af0400a27c12957dcc91259106b40cbd871a02ad5c3" already present on machine | |
kserve-ci-e2e-test |
multus |
model-chainer-raw-hpa-48552-86d85b86d-lrnss |
AddedInterface |
Add eth0 [10.133.0.32/23] from ovn-kubernetes | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-48552-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-graph-raw-hpa-48552-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-c61ee-predictor-54fdf9d498 |
SuccessfulCreate |
Created pod: message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-c61ee-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-c61ee-predictor-54fdf9d498 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "message-dumper-raw-c61ee-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-48552-86d85b86d-lrnss |
Killing |
Stopping container model-chainer-raw-hpa-48552 | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn |
Pulling |
Pulling image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn |
AddedInterface |
Add eth0 [10.134.0.30/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.28:8643/healthz": dial tcp 10.134.0.28:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn |
Started |
Started container kube-rbac-proxy | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.28:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn |
Pulled |
Successfully pulled image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" in 1.295s (1.295s including waiting). Image size: 14813193 bytes. | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.27:8643/healthz": dial tcp 10.134.0.27:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.27:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-c61ee |
InferenceServiceReady |
InferenceService [message-dumper-raw-c61ee] is Ready | |
| (x7) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-c61ee |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-42374c-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-42374c-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-c61ee |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-c61ee": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-c61ee": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-c61ee-predictor-6d9cdc4d99 |
SuccessfulCreate |
Created pod: isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm | |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-c61ee-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-c61ee-predictor-6d9cdc4d99 from 0 to 1 | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm |
AddedInterface |
Add eth0 [10.134.0.31/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:6b08d0c6feafdc947d16756a5e87ed5c77db04b24369e3640cb13be40670b327" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm |
Started |
Started container agent | |
| (x6) | kserve-ci-e2e-test |
kubelet |
model-chainer-raw-hpa-48552-86d85b86d-lrnss |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-c61ee-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-c61ee-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-42374c |
InferenceServiceReady |
InferenceService [isvc-primary-42374c] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-primary-42374c |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-c61ee-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-c61ee-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-42374c |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-42374c": the object has been modified; please apply your changes to the latest version and try again | |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-42374c-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-primary-42374c-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-42374c-predictor-86544c997d-ch6dv |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-secondary-42374c-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-42374c |
UpdateFailed |
Failed to update status for InferenceService "isvc-secondary-42374c": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-secondary-42374c": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-secondary-42374c-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-secondary-42374c-predictor-86544c997d from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-secondary-42374c-predictor-86544c997d |
SuccessfulCreate |
Created pod: isvc-secondary-42374c-predictor-86544c997d-ch6dv | |
kserve-ci-e2e-test |
multus |
isvc-secondary-42374c-predictor-86544c997d-ch6dv |
AddedInterface |
Add eth0 [10.133.0.33/23] from ovn-kubernetes | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-42374c-predictor-86544c997d-ch6dv |
Started |
Started container storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-42374c-predictor-86544c997d-ch6dv |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-secondary-42374c-predictor-86544c997d-ch6dv |
Created |
Created container: storage-initializer |
kserve-ci-e2e-test |
kubelet |
isvc-secondary-42374c-predictor-86544c997d-ch6dv |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-secondary-42374c-predictor-86544c997d-ch6dv_kserve-ci-e2e-test(ed2282ce-e0bf-48ce-af2a-d9c497d56398) | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-secondary-42374c |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-42374c-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-secondary-42374c-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-42374c-predictor-798b4f4b89-bb7vt |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-primary-42374c-predictor-798b4f4b89-bb7vt |
Killing |
Stopping container kserve-container | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-primary-42374c-predictor-798b4f4b89-bb7vt |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.29:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-primary-42374c-predictor-798b4f4b89-bb7vt |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.29:8643/healthz": dial tcp 10.134.0.29:8643: connect: connection refused | |
kserve-ci-e2e-test |
deployment-controller |
isvc-init-fail-d41ccc-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-init-fail-d41ccc-predictor-7cc64b44bd from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-d41ccc |
UpdateFailed |
Failed to update status for InferenceService "isvc-init-fail-d41ccc": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-d41ccc": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-d41ccc |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-init-fail-d41ccc": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-init-fail-d41ccc-predictor-7cc64b44bd |
SuccessfulCreate |
Created pod: isvc-init-fail-d41ccc-predictor-7cc64b44bd-2k4qk | |
kserve-ci-e2e-test |
multus |
isvc-init-fail-d41ccc-predictor-7cc64b44bd-2k4qk |
AddedInterface |
Add eth0 [10.133.0.34/23] from ovn-kubernetes | |
| (x12) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-c61ee |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-c61ee |
InferenceServiceReady |
InferenceService [isvc-logger-raw-c61ee] is Ready | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-d41ccc-predictor-7cc64b44bd-2k4qk |
Created |
Created container: storage-initializer |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-d41ccc-predictor-7cc64b44bd-2k4qk |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-init-fail-d41ccc-predictor-7cc64b44bd-2k4qk |
Started |
Started container storage-initializer |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-init-fail-d41ccc |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-init-fail-d41ccc-predictor-7cc64b44bd-2k4qk |
BackOff |
Back-off restarting failed container storage-initializer in pod isvc-init-fail-d41ccc-predictor-7cc64b44bd-2k4qk_kserve-ci-e2e-test(71d9dd9e-e496-40b5-9de2-fc380aa44e2c) | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-c61ee-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-c61ee-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-f0e12-predictor-8656cc4776-v9r9w |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-f0e12-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-f0e12-predictor-8656cc4776 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-f0e12-predictor-8656cc4776 |
SuccessfulCreate |
Created pod: raw-sklearn-f0e12-predictor-8656cc4776-v9r9w | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-f0e12-predictor-8656cc4776-v9r9w |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-f0e12 |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-f0e12": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-f0e12": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-f0e12-predictor-8656cc4776-v9r9w |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine | |
kserve-ci-e2e-test |
multus |
raw-sklearn-f0e12-predictor-8656cc4776-v9r9w |
AddedInterface |
Add eth0 [10.133.0.35/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
deployment-controller |
isvc-sklearn-scale-raw-f447b-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-sklearn-scale-raw-f447b-predictor-9df674fc8 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-sklearn-scale-raw-f447b-predictor-9df674fc8 |
SuccessfulCreate |
Created pod: isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-sklearn-scale-raw-f447b-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g |
AddedInterface |
Add eth0 [10.134.0.32/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-f0e12-predictor-8656cc4776-v9r9w |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-f0e12-predictor-8656cc4776-v9r9w |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-f0e12-predictor-8656cc4776-v9r9w |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-f0e12-predictor-8656cc4776-v9r9w |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-f0e12-predictor-8656cc4776-v9r9w |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-f0e12-predictor-8656cc4776-v9r9w |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g |
Created |
Created container: kube-rbac-proxy | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.31:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |
| (x4) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.31:8643/healthz": dial tcp 10.134.0.31:8643: connect: connection refused |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-f0e12-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-f0e12-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-f447b-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x2) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-f447b-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-f0e12 |
InferenceServiceReady |
InferenceService [raw-sklearn-f0e12] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-f0e12 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-f0e12-predictor-8656cc4776-v9r9w |
Killing |
Stopping container kserve-container | |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-f0e12-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x4) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-f0e12-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-f0e12-predictor-8656cc4776-v9r9w |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-f0e12-predictor-8656cc4776-v9r9w |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.35:8643/healthz": dial tcp 10.133.0.35:8643: connect: connection refused | |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-f0e12-predictor-8656cc4776-v9r9w |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.35:8080: connect: connection refused |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-60bd0 |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-60bd0": the object has been modified; please apply your changes to the latest version and try again | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-60bd0 |
UpdateFailed |
Failed to update status for InferenceService "raw-sklearn-runtime-60bd0": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "raw-sklearn-runtime-60bd0": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "raw-sklearn-runtime-60bd0-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
raw-sklearn-runtime-60bd0-predictor-7597559d5b |
SuccessfulCreate |
Created pod: raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2 | |
kserve-ci-e2e-test |
deployment-controller |
raw-sklearn-runtime-60bd0-predictor |
ScalingReplicaSet |
Scaled up replica set raw-sklearn-runtime-60bd0-predictor-7597559d5b from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine | |
kserve-ci-e2e-test |
multus |
raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2 |
AddedInterface |
Add eth0 [10.134.0.33/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" already present on machine | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-60bd0-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-60bd0-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
| (x8) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-f447b-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x8) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-sklearn-scale-raw-f447b-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-f447b |
InferenceServiceReady |
InferenceService [isvc-sklearn-scale-raw-f447b] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-sklearn-scale-raw-f447b |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-predictor-8cb968449 |
SuccessfulCreate |
Created pod: isvc-canary-create-predictor-8cb968449-cxzjv | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7768896644-v24qh |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-hkr9h |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-create-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-predictor-8cb968449 |
SuccessfulCreate |
Created pod: isvc-canary-create-predictor-8cb968449-hkr9h | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-cxzjv |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-create-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-v2-predictor-7768896644 |
SuccessfulCreate |
Created pod: isvc-canary-create-v2-predictor-7768896644-v24qh | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7768896644-v24qh |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7768896644-v24qh |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-v2-predictor-7768896644-v24qh |
AddedInterface |
Add eth0 [10.134.0.34/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-create-predictor-8cb968449 from 0 to 2 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-create-v2-predictor-7768896644 from 0 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-cxzjv |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-predictor-8cb968449-cxzjv |
AddedInterface |
Add eth0 [10.133.0.36/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-hkr9h |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-cxzjv |
Started |
Started container storage-initializer | |
| (x18) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.32:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-hkr9h |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-hkr9h |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-create-predictor-8cb968449-hkr9h |
AddedInterface |
Add eth0 [10.134.0.35/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-cxzjv |
Created |
Created container: storage-initializer | |
| (x2) | kserve-ci-e2e-test |
kubelet |
isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.32:8643/healthz": dial tcp 10.134.0.32:8643: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7768896644-v24qh |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7768896644-v24qh |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7768896644-v24qh |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7768896644-v24qh |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-cxzjv |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-hkr9h |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-cxzjv |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-cxzjv |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-cxzjv |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-cxzjv |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-hkr9h |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-cxzjv |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7768896644-v24qh |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-hkr9h |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7768896644-v24qh |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-hkr9h |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-hkr9h |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-hkr9h |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-60bd0 |
InferenceServiceReady |
InferenceService [raw-sklearn-runtime-60bd0] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
raw-sklearn-runtime-60bd0 |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-60bd0-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x5) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
raw-sklearn-runtime-60bd0-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2 |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2 |
Killing |
Stopping container kserve-container | |
| (x10) | kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.33:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.33:8643/healthz": dial tcp 10.134.0.33:8643: connect: connection refused | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-hkr9h |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.35:8080: connect: connection refused |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-cxzjv |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.36:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-hkr9h |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-create-predictor |
ScalingReplicaSet |
Scaled down replica set isvc-canary-create-predictor-8cb968449 from 2 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-create-predictor-8cb968449 |
SuccessfulDelete |
Deleted pod: isvc-canary-create-predictor-8cb968449-hkr9h | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-hkr9h |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.35:8643/healthz": dial tcp 10.134.0.35:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-hkr9h |
Killing |
Stopping container kube-rbac-proxy | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-create": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-create": the object has been modified; please apply your changes to the latest version and try again |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
InferenceServiceReady |
InferenceService [isvc-canary-create] is Ready | |
| (x21) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-create |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-cxzjv |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-cxzjv |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7768896644-v24qh |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7768896644-v24qh |
Killing |
Stopping container kserve-container | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7768896644-v24qh |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.34:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-v2-predictor-7768896644-v24qh |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.34:8643/healthz": dial tcp 10.134.0.34:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-create-predictor-8cb968449-cxzjv |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.36:8643/healthz": dial tcp 10.133.0.36:8643: connect: connection refused | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-v2-predictor-65744c97b6 from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-predictor-8bc5b857 |
SuccessfulCreate |
Created pod: isvc-canary-promote-predictor-8bc5b857-9fh8b | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-2gm85 |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "isvc-canary-promote-v2-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-predictor-8bc5b857 from 0 to 2 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-v2-predictor-65744c97b6 |
SuccessfulCreate |
Created pod: isvc-canary-promote-v2-predictor-65744c97b6-2gm85 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-predictor-8bc5b857 |
SuccessfulCreate |
Created pod: isvc-canary-promote-predictor-8bc5b857-gkln2 | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-v2-predictor-65744c97b6-2gm85 |
AddedInterface |
Add eth0 [10.134.0.36/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-2gm85 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-predictor-8bc5b857-9fh8b |
AddedInterface |
Add eth0 [10.134.0.37/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-2gm85 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-9fh8b |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-9fh8b |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-9fh8b |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-2gm85 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-predictor-8bc5b857-gkln2 |
AddedInterface |
Add eth0 [10.133.0.37/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-gkln2 |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-gkln2 |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-gkln2 |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-gkln2 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-9fh8b |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-9fh8b |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-9fh8b |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-9fh8b |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-9fh8b |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-9fh8b |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-gkln2 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-2gm85 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-2gm85 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-2gm85 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-gkln2 |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-gkln2 |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-gkln2 |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-gkln2 |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-2gm85 |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-2gm85 |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-2gm85 |
Created |
Created container: kserve-container | |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-9fh8b |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.37:8080: connect: connection refused |
| (x9) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-gkln2 |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.37:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-gkln2 |
Killing |
Stopping container kube-rbac-proxy | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InternalError |
fails to update InferenceService status: Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
UpdateFailed |
Failed to update status for InferenceService "isvc-canary-promote": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-canary-promote": the object has been modified; please apply your changes to the latest version and try again |
| (x21) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-gkln2 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-predictor-8bc5b857 |
SuccessfulDelete |
Deleted pod: isvc-canary-promote-predictor-8bc5b857-gkln2 | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-predictor |
ScalingReplicaSet |
Scaled down replica set isvc-canary-promote-predictor-8bc5b857 from 2 to 1 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-gkln2 |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.37:8643/healthz": dial tcp 10.133.0.37:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-9fh8b |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
deployment-controller |
isvc-canary-promote-v2-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-canary-promote-v2-predictor-65744c97b6 from 1 to 2 | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-9fh8b |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-canary-promote-v2-predictor-65744c97b6 |
SuccessfulCreate |
Created pod: isvc-canary-promote-v2-predictor-65744c97b6-v7xcp | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-v7xcp |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine | |
kserve-ci-e2e-test |
multus |
isvc-canary-promote-v2-predictor-65744c97b6-v7xcp |
AddedInterface |
Add eth0 [10.133.0.38/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-v7xcp |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-v7xcp |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-predictor-8bc5b857-9fh8b |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.37:8643/healthz": dial tcp 10.134.0.37:8643: connect: connection refused | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-v7xcp |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-v7xcp |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-v7xcp |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-v7xcp |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-v7xcp |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-v7xcp |
Started |
Started container kserve-container | |
| (x2) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-canary-promote |
InferenceServiceReady |
InferenceService [isvc-canary-promote] is Ready |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-v7xcp |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-v7xcp |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-2gm85 |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-2gm85 |
Killing |
Stopping container kube-rbac-proxy | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-v7xcp |
Unhealthy |
Readiness probe failed: dial tcp 10.133.0.38:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-v7xcp |
Unhealthy |
Readiness probe failed: Get "https://10.133.0.38:8643/healthz": dial tcp 10.133.0.38:8643: connect: connection refused | |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-2gm85 |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.36:8080: connect: connection refused |
kserve-ci-e2e-test |
kubelet |
isvc-canary-promote-v2-predictor-65744c97b6-2gm85 |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.36:8643/healthz": dial tcp 10.134.0.36:8643: connect: connection refused | |
kserve-ci-e2e-test |
replicaset-controller |
message-dumper-raw-cipn-predictor-9774bf55d |
SuccessfulCreate |
Created pod: message-dumper-raw-cipn-predictor-9774bf55d-8dskw | |
kserve-ci-e2e-test |
deployment-controller |
message-dumper-raw-cipn-predictor |
ScalingReplicaSet |
Scaled up replica set message-dumper-raw-cipn-predictor-9774bf55d from 0 to 1 | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-cipn |
UpdateFailed |
Failed to update status for InferenceService "message-dumper-raw-cipn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "message-dumper-raw-cipn": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-8dskw |
FailedMount |
MountVolume.SetUp failed for volume "proxy-tls" : secret "message-dumper-raw-cipn-predictor-serving-cert" not found | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-8dskw |
Pulled |
Container image "gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display" already present on machine | |
kserve-ci-e2e-test |
multus |
message-dumper-raw-cipn-predictor-9774bf55d-8dskw |
AddedInterface |
Add eth0 [10.134.0.38/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-8dskw |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-8dskw |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-8dskw |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-8dskw |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-8dskw |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-cipn |
InferenceServiceReady |
InferenceService [message-dumper-raw-cipn] is Ready | |
| (x7) | kserve-ci-e2e-test |
v1beta1Controllers |
message-dumper-raw-cipn |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-cipn-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-cipn-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API | |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-cipn |
UpdateFailed |
Failed to update status for InferenceService "isvc-logger-raw-cipn": Operation cannot be fulfilled on inferenceservices.serving.kserve.io "isvc-logger-raw-cipn": the object has been modified; please apply your changes to the latest version and try again | |
kserve-ci-e2e-test |
deployment-controller |
isvc-logger-raw-cipn-predictor |
ScalingReplicaSet |
Scaled up replica set isvc-logger-raw-cipn-predictor-64c6cf64bf from 0 to 1 | |
kserve-ci-e2e-test |
replicaset-controller |
isvc-logger-raw-cipn-predictor-64c6cf64bf |
SuccessfulCreate |
Created pod: isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q |
Started |
Started container storage-initializer | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q |
Created |
Created container: storage-initializer | |
kserve-ci-e2e-test |
multus |
isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q |
AddedInterface |
Add eth0 [10.134.0.39/23] from ovn-kubernetes | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q |
Pulled |
Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8abde4da7e87a625b928a82fb491989fd006b5b2f5e4745e1871a977e20e0f2e" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q |
Created |
Created container: kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q |
Pulled |
Container image "quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1955" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q |
Started |
Started container kserve-container | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q |
Pulled |
Container image "quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q |
Created |
Created container: kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q |
Started |
Started container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q |
Pulled |
Container image "quay.io/opendatahub/kserve-agent@sha256:6b08d0c6feafdc947d16756a5e87ed5c77db04b24369e3640cb13be40670b327" already present on machine | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q |
Created |
Created container: agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q |
Started |
Started container agent | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-cipn-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) | |
kserve-ci-e2e-test |
horizontal-pod-autoscaler |
message-dumper-raw-cipn-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) | |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-cipn-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-cipn-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API |
kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-cipn |
InferenceServiceReady |
InferenceService [isvc-logger-raw-cipn] is Ready | |
| (x11) | kserve-ci-e2e-test |
v1beta1Controllers |
isvc-logger-raw-cipn |
VirtualServiceCRDNotFound |
Istio VirtualService CRD not present; VirtualService reconciliation skipped. If you do not use Istio, set ingress.disableIstioVirtualHost=true. |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-cipn-predictor |
FailedComputeMetricsReplicas |
invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
| (x3) | kserve-ci-e2e-test |
horizontal-pod-autoscaler |
isvc-logger-raw-cipn-predictor |
FailedGetResourceMetric |
failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready) |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-8dskw |
Killing |
Stopping container kube-rbac-proxy | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q |
Killing |
Stopping container agent | |
kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q |
Killing |
Stopping container kserve-container | |
kserve-ci-e2e-test |
kubelet |
message-dumper-raw-cipn-predictor-9774bf55d-8dskw |
Killing |
Stopping container kserve-container | |
| (x10) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q |
Unhealthy |
Readiness probe failed: dial tcp 10.134.0.39:8080: connect: connection refused |
| (x4) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q |
Unhealthy |
Readiness probe failed: Get "https://10.134.0.39:8643/healthz": dial tcp 10.134.0.39:8643: connect: connection refused |
| (x11) | kserve-ci-e2e-test |
kubelet |
isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q |
Unhealthy |
Readiness probe failed: HTTP probe failed with statuscode: 503 |